[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:18 dist-test-slave-5hd1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:18 dist-test-slave-5hd1 krb5kdc[399](info): set up 2 sockets
Sep 29 14:47:18 dist-test-slave-5hd1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 14:47:19.604658 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:19 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:19.732055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40799
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:19.842638 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:19.842968 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:19.843041 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:19.847172 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:19.847267 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:19.847307 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:19.847373 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:19.847451 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:19.853199 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:19.854302 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:19.855374 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:47:19.864295 412 server_base.cc:1034] running on GCE node
W20260929 14:47:19.864351 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:19.864384 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:19.864346 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:19.865097 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:19.866151 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:19.867347 412 hybrid_clock.cc:648] HybridClock initialized: now 1790693239867302 us; error 56 us; skew 500 ppm
Sep 29 14:47:19 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:19.870121 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:19.870991 412 webserver.cc:460] Webserver started at http://127.0.98.62:44319/ using document root <none> and password file <none>
I20260929 14:47:19.871495 412 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:19.871551 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:19.871708 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:19.873732 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "d578d2be6c3d400fa278fd122570c593"
format_stamp: "Formatted at 2026-09-29 14:47:19 on dist-test-slave-5hd1"
server_key: "4fb0c68df0024a506b6a61fb2e0af8f0"
server_key_iv: "bc79bc834637f6e420c12d4ee84b6295"
server_key_version: "encryptionkey@0"
I20260929 14:47:19.874231 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "d578d2be6c3d400fa278fd122570c593"
format_stamp: "Formatted at 2026-09-29 14:47:19 on dist-test-slave-5hd1"
server_key: "4fb0c68df0024a506b6a61fb2e0af8f0"
server_key_iv: "bc79bc834637f6e420c12d4ee84b6295"
server_key_version: "encryptionkey@0"
I20260929 14:47:19.878010 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 14:47:19.880366 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:19.881628 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:47:19.881748 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "d578d2be6c3d400fa278fd122570c593"
format_stamp: "Formatted at 2026-09-29 14:47:19 on dist-test-slave-5hd1"
server_key: "4fb0c68df0024a506b6a61fb2e0af8f0"
server_key_iv: "bc79bc834637f6e420c12d4ee84b6295"
server_key_version: "encryptionkey@0"
I20260929 14:47:19.881848 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:19.906714 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:19.909654 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:19.909868 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:19.917027 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40799
I20260929 14:47:19.917040 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40799 every 8 connection(s)
I20260929 14:47:19.918084 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:19.920876 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:19.926062 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593: Bootstrap starting.
I20260929 14:47:19.928318 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:19.928694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 412
I20260929 14:47:19.928846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:19.929122 480 log.cc:824] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:19.930267 392 external_mini_cluster.cc:1428] Setting key 659aeca7da28607a41404bd10420d2da
I20260929 14:47:19.931023 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593: No bootstrap required, opened a new log
I20260929 14:47:19.935514 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } }
I20260929 14:47:19.935752 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:19.935837 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d578d2be6c3d400fa278fd122570c593, State: Initialized, Role: FOLLOWER
I20260929 14:47:19.936307 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } }
I20260929 14:47:19.936461 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:19.936548 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:19.936642 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:19.937474 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } }
I20260929 14:47:19.937812 480 leader_election.cc:302] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d578d2be6c3d400fa278fd122570c593; no voters:
I20260929 14:47:19.938124 480 leader_election.cc:288] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:19.938218 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:19.938376 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [term 1 LEADER]: Becoming Leader. State: Replica: d578d2be6c3d400fa278fd122570c593, State: Running, Role: LEADER
I20260929 14:47:19.938647 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } }
I20260929 14:47:19.939328 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:19.939766 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d578d2be6c3d400fa278fd122570c593. Latest consensus state: current_term: 1 leader_uuid: "d578d2be6c3d400fa278fd122570c593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } } }
I20260929 14:47:19.939875 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:19.940284 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:19.942167 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d578d2be6c3d400fa278fd122570c593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d578d2be6c3d400fa278fd122570c593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40799 } } }
I20260929 14:47:19.942278 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:19.943466 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:19.948915 489 catalog_manager.cc:1347] Generated new cluster ID: 33d6af2098a6419bbe891241ff7e3397
I20260929 14:47:19.949019 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 14:47:19 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:19.955973 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:19.956696 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:19.933067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.62:40799)
0929 14:47:19.933486 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:19.933501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:19.934757 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 14:47:19.937716 (+ 2959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:19.937729 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:19.938113 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:19.938490 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:19.942465 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:19.943549 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:19.944935 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:19.945462 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:19.945623 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:19.950457 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:19.950463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:19.950478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:19.950507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:19.953031 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:19.953660 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:19.953667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:19.953672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:19.953816 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:19.954207 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:19.954211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:19.954213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:19.954603 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:19.955268 (+ 665us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:19.955430 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 14:47:19.955718 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260929 14:47:19.957088 489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:19.964895 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d578d2be6c3d400fa278fd122570c593: Generated new TSK 0
I20260929 14:47:19.965476 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:19.999305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:20.110380 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:20.110631 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:20.110673 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:20.114696 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:20.114751 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:20.114837 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:20.119903 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:20.120929 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:20.121929 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:20.131493 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:20.131673 506 server_base.cc:1034] running on GCE node
W20260929 14:47:20.131490 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:20.131546 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:20.132344 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:20.133035 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:20.134243 506 hybrid_clock.cc:648] HybridClock initialized: now 1790693240134228 us; error 28 us; skew 500 ppm
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:20.137187 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:20.138000 506 webserver.cc:460] Webserver started at http://127.0.98.1:46001/ using document root <none> and password file <none>
I20260929 14:47:20.138540 506 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:20.138662 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:20.138898 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:20.141215 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f9c7a57e3734788877e4b260a6a6f9e"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "2b0668f7696114acde2e1dab06f9959a"
server_key_iv: "ce9967d51b575e78b5b6cc70b4f16af3"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.141722 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f9c7a57e3734788877e4b260a6a6f9e"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "2b0668f7696114acde2e1dab06f9959a"
server_key_iv: "ce9967d51b575e78b5b6cc70b4f16af3"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.145489 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:20.147864 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.149186 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:20.149308 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "5f9c7a57e3734788877e4b260a6a6f9e"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "2b0668f7696114acde2e1dab06f9959a"
server_key_iv: "ce9967d51b575e78b5b6cc70b4f16af3"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.149417 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:20.168118 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:20.170989 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:20.171185 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:20.171752 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:20.172678 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:20.172750 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.172817 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:20.172860 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.182870 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33331
I20260929 14:47:20.182894 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33331 every 8 connection(s)
I20260929 14:47:20.183770 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:20.185662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 506
I20260929 14:47:20.185799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:20.186033 392 external_mini_cluster.cc:1428] Setting key 012c42dd434b3e86f40437812cd3bfb0
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:20.199796 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.185644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39201 (local address 127.0.98.62:40799)
0929 14:47:20.185804 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:20.185812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:20.186767 (+ 955us) server_negotiation.cc:407] Connection header received
0929 14:47:20.189589 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:20.189595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:20.189666 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:20.189765 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:20.190907 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.191797 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.193008 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.193455 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.193589 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.196240 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:20.196246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:20.196251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:20.196286 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:20.197676 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.198230 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.198233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.198234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.198302 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.198582 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.198584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.198585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.198809 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:20.198910 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:20.199483 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 14:47:20.199617 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:47:20.200639 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.186103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40799 (local address 127.0.98.1:39201)
0929 14:47:20.186572 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:20.186652 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:20.188960 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:20.189942 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:20.189949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:20.190296 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:20.190761 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.190774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.191933 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.191937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.192882 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.192891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.193618 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.193621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.193757 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.194367 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:20.194383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:20.195968 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:20.197778 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.197783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.197794 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.198148 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.198377 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.198379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.198381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.198504 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.198969 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:20.198976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:20.199245 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 14:47:20.199438 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 14:47:20.199756 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260929 14:47:20.201913 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40799
I20260929 14:47:20.202157 635 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:20.202777 635 heartbeater.cc:511] Master 127.0.98.62:40799 requested a full tablet report, sending...
I20260929 14:47:20.204629 444 ts_manager.cc:194] Registered new tserver with Master: 5f9c7a57e3734788877e4b260a6a6f9e (127.0.98.1:33331)
I20260929 14:47:20.206305 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:20.216451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:20.326076 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:20.326361 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:20.326407 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:20.330782 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:20.330847 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:20.330940 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:20.336333 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:20.337391 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:20.338478 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:20.348330 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:20.348404 640 server_base.cc:1034] running on GCE node
W20260929 14:47:20.348341 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:20.348348 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:20.348892 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:20.349493 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:20.350665 640 hybrid_clock.cc:648] HybridClock initialized: now 1790693240350640 us; error 36 us; skew 500 ppm
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:20.353530 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:20.354393 640 webserver.cc:460] Webserver started at http://127.0.98.2:37967/ using document root <none> and password file <none>
I20260929 14:47:20.354928 640 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:20.354985 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:20.355144 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:20.357148 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "31e9d6f90434467fbb0c695c99b9bc05"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "5331a2e13fa436d8b25f3e62a4487cae"
server_key_iv: "1a15c30ad417724dd8d50722e893946e"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.357640 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31e9d6f90434467fbb0c695c99b9bc05"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "5331a2e13fa436d8b25f3e62a4487cae"
server_key_iv: "1a15c30ad417724dd8d50722e893946e"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.361140 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:20.363394 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.364631 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:20.364722 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "31e9d6f90434467fbb0c695c99b9bc05"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "5331a2e13fa436d8b25f3e62a4487cae"
server_key_iv: "1a15c30ad417724dd8d50722e893946e"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.364809 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:20.378763 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:20.382306 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:20.382535 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:20.383170 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:20.383940 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:20.383986 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.384024 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:20.384047 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.393249 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36713
I20260929 14:47:20.393273 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36713 every 8 connection(s)
I20260929 14:47:20.394086 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:20.402909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 640
I20260929 14:47:20.403005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:20.403324 392 external_mini_cluster.cc:1428] Setting key 791b88cb158e1cf2987514488e625684
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:20.408486 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.395753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60383 (local address 127.0.98.62:40799)
0929 14:47:20.395886 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:20.395890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:20.396747 (+ 857us) server_negotiation.cc:407] Connection header received
0929 14:47:20.398075 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:20.398078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:20.398136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:20.398223 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:20.399376 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.399922 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.401077 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.401307 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.401419 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.404387 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:20.404390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:20.404392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:20.404415 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:20.406117 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.406640 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.406645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.406649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.406749 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.407204 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.407207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.407208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.407415 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:20.407510 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:20.408256 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 14:47:20.408368 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:47:20.409634 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.396117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40799 (local address 127.0.98.2:60383)
0929 14:47:20.396596 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:20.396641 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:20.397857 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:20.398409 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:20.398416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:20.398763 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:20.399223 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.399236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.400048 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.400052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.400945 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.400954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.401452 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.401457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.401626 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.402543 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:20.402559 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:20.404229 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:20.406230 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.406235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.406247 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.406493 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.406849 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.406853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.406858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.407051 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.407532 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:20.407541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:20.407898 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 14:47:20.408101 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 14:47:20.408473 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
I20260929 14:47:20.410930 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40799
I20260929 14:47:20.411190 769 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:20.411679 769 heartbeater.cc:511] Master 127.0.98.62:40799 requested a full tablet report, sending...
I20260929 14:47:20.412730 444 ts_manager.cc:194] Registered new tserver with Master: 31e9d6f90434467fbb0c695c99b9bc05 (127.0.98.2:36713)
I20260929 14:47:20.413297 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60383
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:20.432006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:20.542532 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:20.542817 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:20.542902 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:20.547066 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:20.547142 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:20.547245 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:20.555510 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:20.557107 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:20.558537 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:20.568377 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:20.568341 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:20.568332 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:20.568703 774 server_base.cc:1034] running on GCE node
I20260929 14:47:20.569147 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:20.569784 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:20.570978 774 hybrid_clock.cc:648] HybridClock initialized: now 1790693240570965 us; error 27 us; skew 500 ppm
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:20.573814 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:20.574782 774 webserver.cc:460] Webserver started at http://127.0.98.3:36747/ using document root <none> and password file <none>
I20260929 14:47:20.575364 774 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:20.575457 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:20.575686 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:20.577827 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f2e81a9f0254b3983367bfd3c8d1df3"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "d58eb53c5e0c284192cce07c62e9e270"
server_key_iv: "67adddff2dc17cd237d1d197c3918163"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.578387 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f2e81a9f0254b3983367bfd3c8d1df3"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "d58eb53c5e0c284192cce07c62e9e270"
server_key_iv: "67adddff2dc17cd237d1d197c3918163"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.582494 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:47:20.584933 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.586243 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:20.586373 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "3f2e81a9f0254b3983367bfd3c8d1df3"
format_stamp: "Formatted at 2026-09-29 14:47:20 on dist-test-slave-5hd1"
server_key: "d58eb53c5e0c284192cce07c62e9e270"
server_key_iv: "67adddff2dc17cd237d1d197c3918163"
server_key_version: "encryptionkey@0"
I20260929 14:47:20.586483 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:20.606398 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:20.609649 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:20.609872 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:20.610526 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:20.611382 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:20.611454 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.611529 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:20.611573 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:20.621865 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41875
I20260929 14:47:20.621886 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41875 every 8 connection(s)
I20260929 14:47:20.622776 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:20.628233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 774
I20260929 14:47:20.628353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:20.628630 392 external_mini_cluster.cc:1428] Setting key ffa49f167426026bb8e6ca5648c3c85a
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:20.637405 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.624729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37537 (local address 127.0.98.62:40799)
0929 14:47:20.624910 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:20.624914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:20.625718 (+ 804us) server_negotiation.cc:407] Connection header received
0929 14:47:20.627070 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:20.627074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:20.627128 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:20.627236 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:20.628690 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.629240 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.630771 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.631036 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.631145 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.633777 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:20.633781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:20.633783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:20.633806 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:20.635137 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.635640 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.635642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.635643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.635716 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.636066 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.636069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.636070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.636535 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:20.636620 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:20.637177 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 14:47:20.637276 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:47:20.638259 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.625103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40799 (local address 127.0.98.3:37537)
0929 14:47:20.625562 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:20.625606 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:20.626857 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:20.627414 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:20.627427 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:20.627784 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:20.628511 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.628532 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.629364 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.629368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.630647 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:20.630656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.631193 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.631196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:20.631351 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.632064 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:20.632081 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:20.633617 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:20.635240 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.635245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.635257 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.635550 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.635828 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:20.635831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:20.635832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:20.635971 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:20.636683 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:20.636688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:20.636933 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 14:47:20.637102 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 14:47:20.637360 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260929 14:47:20.639559 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40799
I20260929 14:47:20.639894 903 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:20.640471 903 heartbeater.cc:511] Master 127.0.98.62:40799 requested a full tablet report, sending...
I20260929 14:47:20.641538 444 ts_manager.cc:194] Registered new tserver with Master: 3f2e81a9f0254b3983367bfd3c8d1df3 (127.0.98.3:41875)
I20260929 14:47:20.642293 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37537
I20260929 14:47:20.643181 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:20.670127 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.660605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.0.98.1:33331)
0929 14:47:20.660884 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:20.660901 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:20.660931 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:47:20.661113 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:20.661119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:20.661307 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:20.661457 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:20.662113 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.662999 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.663842 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.664273 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.664388 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.666701 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:20.666709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:20.666721 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:20.666745 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:20.668382 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.668807 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.668813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.668814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.668891 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.669173 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.669176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.669178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.669409 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:20.669643 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:20.669802 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 14:47:20.669955 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
W20260929 14:47:20.671030 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:20.697264 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:20.689125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.0.98.1:33331)
0929 14:47:20.689312 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:20.689316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:20.689336 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:47:20.689474 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:20.689477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:20.689526 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:20.689600 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:20.690352 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.690878 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.691779 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:20.692008 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:20.692086 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:20.694491 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:20.694494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:20.694496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:20.694519 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:20.695782 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.696138 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.696141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.696142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.696275 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:20.696507 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:20.696509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:20.696511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:20.696660 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:20.696755 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:20.696948 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 14:47:20.697046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 14:47:20.698299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 506
I20260929 14:47:20.704162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 640
I20260929 14:47:20.712009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 774
I20260929 14:47:20.716950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 412
2026-09-29T14:47:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2172 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[917](info): set up 2 sockets
Sep 29 14:47:20 dist-test-slave-5hd1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 14:47:21.759793 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:21 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:21.883005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:21.990880 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:21.991165 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:21.991207 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:21.995306 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:21.995359 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:21.995378 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:21.995395 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:21.995440 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:22.000756 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:22.001811 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:22.002859 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:22.011462 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:22.011492 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:22.011555 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.011485 930 server_base.cc:1034] running on GCE node
I20260929 14:47:22.012346 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:22.013429 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:22.014618 930 hybrid_clock.cc:648] HybridClock initialized: now 1790693242014593 us; error 44 us; skew 500 ppm
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:22.017324 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:22.018216 930 webserver.cc:460] Webserver started at http://127.0.98.62:40967/ using document root <none> and password file <none>
I20260929 14:47:22.018769 930 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:22.018858 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:22.019083 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:22.021126 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "c60551116fe739fecf105758d2703e05"
server_key_iv: "bed89b47fa0f296dedc0492f45f546b3"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.021684 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "c60551116fe739fecf105758d2703e05"
server_key_iv: "bed89b47fa0f296dedc0492f45f546b3"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.025341 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:22.027633 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.028899 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:22.029036 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "c60551116fe739fecf105758d2703e05"
server_key_iv: "bed89b47fa0f296dedc0492f45f546b3"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.029145 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:22.050150 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:22.053292 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:22.053504 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:22.060328 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41935
I20260929 14:47:22.060343 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41935 every 8 connection(s)
I20260929 14:47:22.061261 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:22.064005 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:22.068570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 930
I20260929 14:47:22.068660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:22.068912 392 external_mini_cluster.cc:1428] Setting key ec2f7b3b45cd13d4e53a7d72f85a142f
I20260929 14:47:22.069381 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: Bootstrap starting.
I20260929 14:47:22.071877 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:22.072736 998 log.cc:824] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:22.074728 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: No bootstrap required, opened a new log
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:22.078402 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 14:47:22.078680 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:22.078753 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae21fc3bf5fc4bdebbfd812ee67de8b6, State: Initialized, Role: FOLLOWER
I20260929 14:47:22.079303 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 14:47:22.079484 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:22.079558 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:22.079669 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:22.080857 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 14:47:22.081240 998 leader_election.cc:302] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae21fc3bf5fc4bdebbfd812ee67de8b6; no voters:
I20260929 14:47:22.081579 998 leader_election.cc:288] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:22.081696 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:22.081903 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [term 1 LEADER]: Becoming Leader. State: Replica: ae21fc3bf5fc4bdebbfd812ee67de8b6, State: Running, Role: LEADER
I20260929 14:47:22.082265 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 14:47:22.082921 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:22.084153 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae21fc3bf5fc4bdebbfd812ee67de8b6. Latest consensus state: current_term: 1 leader_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260929 14:47:22.084246 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.070307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.98.62:41935)
0929 14:47:22.070703 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.070722 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.070762 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:47:22.071544 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.071568 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.071967 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.072345 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.073004 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.074333 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.075204 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.075860 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.076029 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.078453 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.078460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.078475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.078506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.080570 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.080986 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.080992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.080997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.081132 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.081418 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.081421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.081423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.081981 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.082730 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.082876 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.083192 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260929 14:47:22.084146 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae21fc3bf5fc4bdebbfd812ee67de8b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260929 14:47:22.084307 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:22.084321 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6 [sys.catalog]: This master's current role is: LEADER
W20260929 14:47:22.087755 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:22.087839 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:22.087945 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:22.088869 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:22.094753 1013 catalog_manager.cc:1347] Generated new cluster ID: 8c7bef232164429792457c6cde51493a
I20260929 14:47:22.094825 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:22.112115 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:22.113375 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:22.123857 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae21fc3bf5fc4bdebbfd812ee67de8b6: Generated new TSK 0
I20260929 14:47:22.124588 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:22.161367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:22.273542 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:22.273823 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:22.273869 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:22.277905 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:22.277967 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:22.278028 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:22.283372 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:22.284554 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:22.285609 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:22.295264 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:22.295265 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.295317 1024 server_base.cc:1034] running on GCE node
W20260929 14:47:22.295260 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.295946 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:22.296617 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:22.297822 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790693242297784 us; error 46 us; skew 500 ppm
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:22.300714 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:22.301581 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41405/ using document root <none> and password file <none>
I20260929 14:47:22.302151 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:22.302212 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:22.302451 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:22.304616 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "31d87b8cbc824944a9a3ab65c6edab2d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "31ee80b66739a0d65b0a6af0db1c8427"
server_key_iv: "458cac861c940351a2625e5076d5d32d"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.305186 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31d87b8cbc824944a9a3ab65c6edab2d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "31ee80b66739a0d65b0a6af0db1c8427"
server_key_iv: "458cac861c940351a2625e5076d5d32d"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.308984 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:47:22.311471 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.312812 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:22.312952 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "31d87b8cbc824944a9a3ab65c6edab2d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "31ee80b66739a0d65b0a6af0db1c8427"
server_key_iv: "458cac861c940351a2625e5076d5d32d"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.313068 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:22.343092 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:22.346662 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:22.346894 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:22.347525 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:22.348467 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:22.348541 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.348614 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:22.348657 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.357712 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41333
I20260929 14:47:22.357729 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41333 every 8 connection(s)
I20260929 14:47:22.358637 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:22.368270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1024
I20260929 14:47:22.368396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:22.368649 392 external_mini_cluster.cc:1428] Setting key 1bc4aa9c4d138afc712040daf136ae0d
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:22.373499 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.360412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58283 (local address 127.0.98.62:41935)
0929 14:47:22.360555 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.360560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.361471 (+ 911us) server_negotiation.cc:407] Connection header received
0929 14:47:22.362873 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.362877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.362948 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.363058 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.364696 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.365354 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.366568 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.366815 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.366937 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.369567 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.369571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.369573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.369598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.371095 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.371770 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.371775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.371778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.371887 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.372275 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.372281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.372283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.372529 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.372637 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.373121 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.373265 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:47:22.374163 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.360816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.98.1:58283)
0929 14:47:22.361271 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.361316 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.362644 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.363255 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.363262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.363662 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:22.364497 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.364540 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.365490 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.365493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.366436 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.366445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.366983 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.366987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.367166 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.367727 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:22.367743 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:22.369387 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:22.371215 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.371224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.371242 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.371617 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.372006 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.372008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.372009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.372147 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.372635 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:22.372640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:22.372870 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.373019 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.373276 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":102,"threads_started":1}
I20260929 14:47:22.375373 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41935
I20260929 14:47:22.375646 1153 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:22.376291 1153 heartbeater.cc:511] Master 127.0.98.62:41935 requested a full tablet report, sending...
I20260929 14:47:22.377818 962 ts_manager.cc:194] Registered new tserver with Master: 31d87b8cbc824944a9a3ab65c6edab2d (127.0.98.1:41333)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:22.378970 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:22.396320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:22.504575 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:22.504819 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:22.504861 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:22.508790 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:22.508846 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:22.508903 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:22.513967 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:22.514902 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:22.515820 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:22.525620 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.525691 1158 server_base.cc:1034] running on GCE node
W20260929 14:47:22.525631 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:22.525669 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.526242 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:22.526921 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:22.528141 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790693242528111 us; error 37 us; skew 500 ppm
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:22.531139 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:22.532136 1158 webserver.cc:460] Webserver started at http://127.0.98.2:34209/ using document root <none> and password file <none>
I20260929 14:47:22.532822 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:22.532918 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:22.533155 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:22.535240 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "f41f4bf1652645fb8590e9db873f4e55"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "9a07ccb4020451339cf42258b1f62d02"
server_key_iv: "88edee1eb5c81c24af711f498af8dbbb"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.535794 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f41f4bf1652645fb8590e9db873f4e55"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "9a07ccb4020451339cf42258b1f62d02"
server_key_iv: "88edee1eb5c81c24af711f498af8dbbb"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.539541 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 14:47:22.542276 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.543709 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:22.543870 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "f41f4bf1652645fb8590e9db873f4e55"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "9a07ccb4020451339cf42258b1f62d02"
server_key_iv: "88edee1eb5c81c24af711f498af8dbbb"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.543998 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:22.558473 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:22.561743 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:22.561952 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:22.562546 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:22.563392 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:22.563463 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.563529 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:22.563583 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.572465 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41867
I20260929 14:47:22.572484 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41867 every 8 connection(s)
I20260929 14:47:22.573362 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:22.582465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1158
I20260929 14:47:22.582556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:22.582849 392 external_mini_cluster.cc:1428] Setting key b02de69e282e7b19b6de08729bdc0728
I20260929 14:47:22.587455 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.575156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58361 (local address 127.0.98.62:41935)
0929 14:47:22.575287 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.575290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.576106 (+ 816us) server_negotiation.cc:407] Connection header received
0929 14:47:22.577420 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.577423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.577467 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.577540 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.578679 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.579191 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.580360 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.580563 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.580649 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.583251 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.583254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.583256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.583301 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.585080 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.585870 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.585873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.585874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.585947 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.586363 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.586368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.586370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.586562 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.586670 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.587182 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.587298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:47:22.588371 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.575542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.98.2:58361)
0929 14:47:22.575954 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.575995 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.577233 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.577721 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.577728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.578083 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:22.578551 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.578563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.579291 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.579294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.580261 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.580271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.580709 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.580712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.580855 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.581488 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:22.581504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:22.583072 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:22.585352 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.585361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.585379 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.585757 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.586041 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.586046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.586049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.586252 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.586679 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:22.586688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:22.586937 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.587141 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.587475 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260929 14:47:22.589545 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41935
I20260929 14:47:22.589795 1287 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:22.590292 1287 heartbeater.cc:511] Master 127.0.98.62:41935 requested a full tablet report, sending...
I20260929 14:47:22.591346 962 ts_manager.cc:194] Registered new tserver with Master: f41f4bf1652645fb8590e9db873f4e55 (127.0.98.2:41867)
I20260929 14:47:22.591912 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58361
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:22.610610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:22.719188 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:22.719456 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:22.719537 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:22.723546 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:22.723634 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:22.723726 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:22.728868 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:22.729857 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:22.730842 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:22.740392 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:22.740537 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.740437 1292 server_base.cc:1034] running on GCE node
W20260929 14:47:22.740406 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:22.741034 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:22.741613 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:22.742770 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790693242742738 us; error 46 us; skew 500 ppm
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:22.745642 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:22.746498 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38837/ using document root <none> and password file <none>
I20260929 14:47:22.747159 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:22.747253 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:22.747460 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:22.749640 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "429b2cc194764e4783c1c195bad19f3d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "d29deecfd72e80fff238853f6ea9c905"
server_key_iv: "b2a9c6f209a591d347b1319ad047f209"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.750216 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "429b2cc194764e4783c1c195bad19f3d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "d29deecfd72e80fff238853f6ea9c905"
server_key_iv: "b2a9c6f209a591d347b1319ad047f209"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.754015 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:47:22.756580 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.757813 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:47:22.757962 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "429b2cc194764e4783c1c195bad19f3d"
format_stamp: "Formatted at 2026-09-29 14:47:22 on dist-test-slave-5hd1"
server_key: "d29deecfd72e80fff238853f6ea9c905"
server_key_iv: "b2a9c6f209a591d347b1319ad047f209"
server_key_version: "encryptionkey@0"
I20260929 14:47:22.758081 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:22.792922 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:22.795854 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:22.796061 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:22.796707 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:22.797519 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:22.797601 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.797676 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:22.797710 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:22.806013 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46247
I20260929 14:47:22.806044 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46247 every 8 connection(s)
I20260929 14:47:22.806982 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:22.816799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1292
I20260929 14:47:22.816920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:22.817214 392 external_mini_cluster.cc:1428] Setting key f8b7c4e5fd04aad5d812af154483e32f
I20260929 14:47:22.820925 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.808838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41365 (local address 127.0.98.62:41935)
0929 14:47:22.808969 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.808973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.809847 (+ 874us) server_negotiation.cc:407] Connection header received
0929 14:47:22.811306 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.811309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.811368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.811466 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.812695 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.813231 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.814359 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.814583 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.814694 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.817274 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.817278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.817282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.817310 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.818938 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.819522 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.819527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.819529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.819616 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.819908 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.819910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.819911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.820081 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.820153 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.820719 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.820815 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:47:22.821789 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.809263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.98.3:41365)
0929 14:47:22.809704 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.809745 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.811094 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.811639 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.811645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.812006 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:22.812552 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.812572 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.813352 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.813358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.814235 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.814244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.814730 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.814735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.814898 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.815491 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:22.815507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:22.817116 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:22.819043 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.819048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.819063 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.819414 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.819701 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:22.819703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:22.819704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:22.819832 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:22.820165 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:22.820169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:22.820500 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.820664 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.820913 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260929 14:47:22.822875 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41935
I20260929 14:47:22.823129 1421 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:22.823597 1421 heartbeater.cc:511] Master 127.0.98.62:41935 requested a full tablet report, sending...
I20260929 14:47:22.824630 962 ts_manager.cc:194] Registered new tserver with Master: 429b2cc194764e4783c1c195bad19f3d (127.0.98.3:46247)
I20260929 14:47:22.825181 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41365
I20260929 14:47:22.831103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:47:22.841776 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.834518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.98.62:41935)
0929 14:47:22.834646 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.834650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.834799 (+ 149us) server_negotiation.cc:407] Connection header received
0929 14:47:22.834890 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.834892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.834927 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.834990 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.835658 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.836241 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.837045 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.837252 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.837332 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.838690 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.838693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.838695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.838715 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.840120 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.840671 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.840674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.840676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.840754 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.841030 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.841032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.841033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.841240 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.841342 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.841564 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.841658 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:47:22.847078 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:22.850268 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:22.864845 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.858843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.0.98.3:46247)
0929 14:47:22.860302 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.860313 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.860330 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:47:22.860386 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.860389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.860523 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.860684 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:22.860976 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.862144 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.863505 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.864144 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864279 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.864414 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.864590 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.864685 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":160,"threads_started":1}
I20260929 14:47:22.864842 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.859067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41867 (local address 127.0.0.1:34524)
0929 14:47:22.859916 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.859932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.860051 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.860577 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.860583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.860604 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:22.860771 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.860779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.861902 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.861908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.863238 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.863248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.864483 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.864577 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.864596 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.864657 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.864706 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260929 14:47:22.864842 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.858717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46247 (local address 127.0.0.1:54840)
0929 14:47:22.860200 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.860213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.860275 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.860663 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.860666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.860688 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:22.860854 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.860862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.862279 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.862282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.863394 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.863401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.864294 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.864399 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.864433 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.864541 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.864674 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":48,"thread_start_us":99,"threads_started":1}
I20260929 14:47:22.865513 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.859402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.0.98.2:41867)
0929 14:47:22.859841 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.859854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.860016 (+ 162us) server_negotiation.cc:407] Connection header received
0929 14:47:22.860220 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.860231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.860433 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.860598 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:22.860881 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.861779 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.863367 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.864138 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864297 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.864434 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.865250 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.865350 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":96,"threads_started":1}
I20260929 14:47:22.866425 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.858743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41333 (local address 127.0.0.1:46604)
0929 14:47:22.859870 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.859929 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.860051 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.861307 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.861310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.861325 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:22.861400 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.861405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.862847 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.862850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.863883 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.863894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.865333 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.865336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.865411 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.865461 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.865525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.866300 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":73,"threads_started":1}
I20260929 14:47:22.866871 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.859553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.98.1:41333)
0929 14:47:22.860083 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.860103 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.860733 (+ 630us) server_negotiation.cc:407] Connection header received
0929 14:47:22.860813 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.860821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.861016 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.861154 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:22.861519 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.862643 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864010 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.864819 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.864926 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.866593 (+ 1667us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.866642 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.866727 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":88,"threads_started":1}
I20260929 14:47:22.867956 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 750fe08670c54a5aa1d7cba58ac23ffb (DEFAULT_TABLE table=test-table [id=2be3fe0302a04daa8d82c86ce181b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:22.868125 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 750fe08670c54a5aa1d7cba58ac23ffb (DEFAULT_TABLE table=test-table [id=2be3fe0302a04daa8d82c86ce181b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:22.869053 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750fe08670c54a5aa1d7cba58ac23ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:22.869077 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750fe08670c54a5aa1d7cba58ac23ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:22.869350 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 750fe08670c54a5aa1d7cba58ac23ffb (DEFAULT_TABLE table=test-table [id=2be3fe0302a04daa8d82c86ce181b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:22.870232 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750fe08670c54a5aa1d7cba58ac23ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:22.875180 1438 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: Bootstrap starting.
I20260929 14:47:22.876681 1439 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: Bootstrap starting.
I20260929 14:47:22.876864 1440 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: Bootstrap starting.
I20260929 14:47:22.877789 1438 tablet_bootstrap.cc:654] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:22.878511 1439 tablet_bootstrap.cc:654] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:22.878912 1440 tablet_bootstrap.cc:654] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:22.879078 1438 log.cc:824] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:22.879647 1439 log.cc:824] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:22.879695 1440 log.cc:824] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:22.881430 1440 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: No bootstrap required, opened a new log
I20260929 14:47:22.881476 1438 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: No bootstrap required, opened a new log
I20260929 14:47:22.881649 1440 ts_tablet_manager.cc:1397] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:47:22.881654 1438 ts_tablet_manager.cc:1397] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 14:47:22.881698 1439 tablet_bootstrap.cc:492] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: No bootstrap required, opened a new log
I20260929 14:47:22.881831 1439 ts_tablet_manager.cc:1397] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:47:22.885285 1440 raft_consensus.cc:348] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.885542 1440 raft_consensus.cc:374] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:22.885597 1440 raft_consensus.cc:729] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d87b8cbc824944a9a3ab65c6edab2d, State: Initialized, Role: FOLLOWER
I20260929 14:47:22.886059 1440 consensus_queue.cc:260] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.886125 1438 raft_consensus.cc:348] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.886124 1439 raft_consensus.cc:348] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.886384 1439 raft_consensus.cc:374] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:22.886384 1438 raft_consensus.cc:374] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:22.886447 1439 raft_consensus.cc:729] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429b2cc194764e4783c1c195bad19f3d, State: Initialized, Role: FOLLOWER
I20260929 14:47:22.886456 1438 raft_consensus.cc:729] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41f4bf1652645fb8590e9db873f4e55, State: Initialized, Role: FOLLOWER
I20260929 14:47:22.886875 1439 consensus_queue.cc:260] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.886875 1438 consensus_queue.cc:260] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.887480 1440 ts_tablet_manager.cc:1428] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:22.887611 1153 heartbeater.cc:503] Master 127.0.98.62:41935 was elected leader, sending a full tablet report...
I20260929 14:47:22.887801 1438 ts_tablet_manager.cc:1428] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 14:47:22.887801 1439 ts_tablet_manager.cc:1428] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:22.888119 1287 heartbeater.cc:503] Master 127.0.98.62:41935 was elected leader, sending a full tablet report...
I20260929 14:47:22.888150 1421 heartbeater.cc:503] Master 127.0.98.62:41935 was elected leader, sending a full tablet report...
I20260929 14:47:22.954594 1444 raft_consensus.cc:482] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:22.954787 1444 raft_consensus.cc:504] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.955896 1444 leader_election.cc:288] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 429b2cc194764e4783c1c195bad19f3d (127.0.98.3:46247), f41f4bf1652645fb8590e9db873f4e55 (127.0.98.2:41867)
I20260929 14:47:22.959729 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.956215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40087 (local address 127.0.98.3:46247)
0929 14:47:22.956366 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.956373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.956539 (+ 166us) server_negotiation.cc:407] Connection header received
0929 14:47:22.956715 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.956717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.956757 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.956835 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:22.957167 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.957661 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.958778 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.959177 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.959293 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.959356 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.959534 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.959600 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:47:22.959743 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.956486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55181 (local address 127.0.98.2:41867)
0929 14:47:22.956607 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.956612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.956626 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:22.956751 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.956754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.956793 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.956874 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:22.957177 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.957657 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.958796 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.959204 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.959319 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.959380 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.959559 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.959621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:47:22.959795 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.956157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41867 (local address 127.0.98.1:55181)
0929 14:47:22.956501 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.956514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.956622 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.956893 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.956896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.956921 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:22.957067 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.957073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.957765 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.957771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.958669 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.958678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.959319 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.959322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.959427 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.959457 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.959544 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.959621 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":55,"thread_start_us":115,"threads_started":1}
I20260929 14:47:22.959802 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.956077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46247 (local address 127.0.98.1:40087)
0929 14:47:22.956417 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:22.956434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:22.956600 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:22.956855 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:22.956858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:22.956874 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:22.957059 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.957065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.957768 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.957774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.958657 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:22.958666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.959295 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.959299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:22.959394 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.959432 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 14:47:22.959519 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:47:22.959589 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":58,"thread_start_us":149,"threads_started":1}
I20260929 14:47:22.960461 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750fe08670c54a5aa1d7cba58ac23ffb" candidate_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429b2cc194764e4783c1c195bad19f3d" is_pre_election: true
I20260929 14:47:22.960520 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750fe08670c54a5aa1d7cba58ac23ffb" candidate_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41f4bf1652645fb8590e9db873f4e55" is_pre_election: true
I20260929 14:47:22.960799 1375 raft_consensus.cc:2454] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d87b8cbc824944a9a3ab65c6edab2d in term 0.
I20260929 14:47:22.960799 1241 raft_consensus.cc:2454] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d87b8cbc824944a9a3ab65c6edab2d in term 0.
I20260929 14:47:22.961334 1043 leader_election.cc:302] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31d87b8cbc824944a9a3ab65c6edab2d, 429b2cc194764e4783c1c195bad19f3d, f41f4bf1652645fb8590e9db873f4e55; no voters:
I20260929 14:47:22.961613 1444 raft_consensus.cc:2781] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:22.961721 1444 raft_consensus.cc:482] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:22.961819 1444 raft_consensus.cc:3037] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:22.962934 1444 raft_consensus.cc:504] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.963289 1444 leader_election.cc:288] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [CANDIDATE]: Term 1 election: Requested vote from peers 429b2cc194764e4783c1c195bad19f3d (127.0.98.3:46247), f41f4bf1652645fb8590e9db873f4e55 (127.0.98.2:41867)
I20260929 14:47:22.963644 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750fe08670c54a5aa1d7cba58ac23ffb" candidate_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41f4bf1652645fb8590e9db873f4e55"
I20260929 14:47:22.963637 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750fe08670c54a5aa1d7cba58ac23ffb" candidate_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429b2cc194764e4783c1c195bad19f3d"
I20260929 14:47:22.963760 1375 raft_consensus.cc:3037] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:22.963761 1241 raft_consensus.cc:3037] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:22.965035 1375 raft_consensus.cc:2454] T 750fe08670c54a5aa1d7cba58ac23ffb P 429b2cc194764e4783c1c195bad19f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d87b8cbc824944a9a3ab65c6edab2d in term 1.
I20260929 14:47:22.965035 1241 raft_consensus.cc:2454] T 750fe08670c54a5aa1d7cba58ac23ffb P f41f4bf1652645fb8590e9db873f4e55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d87b8cbc824944a9a3ab65c6edab2d in term 1.
I20260929 14:47:22.965377 1041 leader_election.cc:302] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d87b8cbc824944a9a3ab65c6edab2d, 429b2cc194764e4783c1c195bad19f3d; no voters:
I20260929 14:47:22.965580 1444 raft_consensus.cc:2781] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:22.965776 1444 raft_consensus.cc:686] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [term 1 LEADER]: Becoming Leader. State: Replica: 31d87b8cbc824944a9a3ab65c6edab2d, State: Running, Role: LEADER
I20260929 14:47:22.966102 1444 consensus_queue.cc:237] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } }
I20260929 14:47:22.969201 962 catalog_manager.cc:5495] T 750fe08670c54a5aa1d7cba58ac23ffb P 31d87b8cbc824944a9a3ab65c6edab2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 31d87b8cbc824944a9a3ab65c6edab2d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "429b2cc194764e4783c1c195bad19f3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d87b8cbc824944a9a3ab65c6edab2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f41f4bf1652645fb8590e9db873f4e55" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41867 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:22.988389 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:22.978783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.0.98.1:41333)
0929 14:47:22.979016 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:22.979020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:22.979145 (+ 125us) server_negotiation.cc:407] Connection header received
0929 14:47:22.979243 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:22.979246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:22.979285 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:22.979356 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:22.980030 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.980641 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.981568 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:22.981784 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:22.981877 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:22.984888 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:22.984906 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:22.984928 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:22.984965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:22.986509 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.986998 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.987012 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.987013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.987093 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:22.987383 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:22.987386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:22.987388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:22.987672 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:22.987911 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:22.988058 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 14:47:22.988217 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
Sep 29 14:47:22 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693242, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:23.015027 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:23.006538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.98.1:41333)
0929 14:47:23.006720 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:23.006724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:23.006851 (+ 127us) server_negotiation.cc:407] Connection header received
0929 14:47:23.007032 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:23.007035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:23.007092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:23.007179 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:23.007852 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:23.008476 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:23.009335 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:23.009567 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:23.009685 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:23.012009 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:23.012012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:23.012014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:23.012038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:23.013341 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:23.013754 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:23.013758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:23.013760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:23.013839 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:23.014129 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:23.014133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:23.014135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:23.014303 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:23.014475 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:23.014738 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 14:47:23.014861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:23.041061 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:23.032262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.0.98.1:41333)
0929 14:47:23.032426 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:23.032431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:23.032629 (+ 198us) server_negotiation.cc:407] Connection header received
0929 14:47:23.032772 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:23.032775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:23.032846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:23.032969 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:23.033669 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:23.034231 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:23.035023 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:23.035241 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:23.035318 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:23.037823 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:23.037826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:23.037828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:23.037869 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:23.039313 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:23.039770 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:23.039776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:23.039778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:23.039865 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:23.040160 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:23.040163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:23.040165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:23.040380 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:23.040542 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:23.040769 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 14:47:23.040892 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 14:47:23.042508 1067 tablet_service.cc:2908] Rejecting scan request for tablet 750fe08670c54a5aa1d7cba58ac23ffb: Uninitialized: clean time has not yet been initialized
I20260929 14:47:23.044687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1024
I20260929 14:47:23.050688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1158
I20260929 14:47:23.056324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1292
I20260929 14:47:23.062107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 930
2026-09-29T14:47:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2344 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[1463](info): set up 2 sockets
Sep 29 14:47:23 dist-test-slave-5hd1 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260929 14:47:24.115927 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:24.244445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:24.354502 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:24.354765 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:24.354805 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:24.358794 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:24.358856 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:47:24.358873 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:24.358891 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:24.358907 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:24.358930 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:24.364117 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33037
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:24.365232 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:24.366219 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:24.374562 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:24.374572 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:24.374568 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:24.374624 1476 server_base.cc:1034] running on GCE node
I20260929 14:47:24.375368 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:24.376463 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:24.377657 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790693244377646 us; error 25 us; skew 500 ppm
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:24.386210 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:24.387187 1476 webserver.cc:460] Webserver started at http://127.0.98.62:46539/ using document root <none> and password file <none>
I20260929 14:47:24.387813 1476 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:24.387908 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:24.388139 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:24.390228 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "1408951d9077499794309cc488320a0a"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "97563ff4b4c66f6c7096a893fd8fe8a4"
server_key_iv: "c924d53976197fcae6ff39921bf13012"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.390820 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "1408951d9077499794309cc488320a0a"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "97563ff4b4c66f6c7096a893fd8fe8a4"
server_key_iv: "c924d53976197fcae6ff39921bf13012"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.394498 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 14:47:24.397179 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:24.398424 1476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:24.398584 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "1408951d9077499794309cc488320a0a"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "97563ff4b4c66f6c7096a893fd8fe8a4"
server_key_iv: "c924d53976197fcae6ff39921bf13012"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.398701 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:24.409754 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:24.412539 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:24.412739 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:24.420161 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33037
I20260929 14:47:24.420197 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33037 every 8 connection(s)
I20260929 14:47:24.421056 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:24.424058 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:24.429224 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Bootstrap starting.
I20260929 14:47:24.430825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1476
I20260929 14:47:24.430935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:24.431200 392 external_mini_cluster.cc:1428] Setting key bd7c15de9eec45465abc82b9d7a5c28e
I20260929 14:47:24.431802 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:24.432723 1554 log.cc:824] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:24.434444 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: No bootstrap required, opened a new log
I20260929 14:47:24.437790 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } }
I20260929 14:47:24.438071 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:24.438154 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1408951d9077499794309cc488320a0a, State: Initialized, Role: FOLLOWER
I20260929 14:47:24.438622 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } }
I20260929 14:47:24.438766 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:24.438834 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:24.438930 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:24.439975 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } }
I20260929 14:47:24.440349 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1408951d9077499794309cc488320a0a; no voters:
I20260929 14:47:24.440678 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:24.440860 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:24.441095 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [term 1 LEADER]: Becoming Leader. State: Replica: 1408951d9077499794309cc488320a0a, State: Running, Role: LEADER
I20260929 14:47:24.441506 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } }
I20260929 14:47:24.441967 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:24.443850 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1408951d9077499794309cc488320a0a. Latest consensus state: current_term: 1 leader_uuid: "1408951d9077499794309cc488320a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } } }
I20260929 14:47:24.444414 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:24.444103 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1408951d9077499794309cc488320a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1408951d9077499794309cc488320a0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33037 } } }
I20260929 14:47:24.444635 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:24.445876 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.432291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.0.98.62:33037)
0929 14:47:24.432725 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:24.432741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:24.432774 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:47:24.433414 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:24.433430 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:24.433733 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:24.434066 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:24.434850 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.435797 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.436690 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.437171 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.437335 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.440161 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:24.440166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:24.440233 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:24.440267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:24.442389 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.442918 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.442927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.442935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.443102 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.443436 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.443440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.443442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.443983 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:24.444226 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:24.444840 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 14:47:24.445092 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
W20260929 14:47:24.447140 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:24.447247 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:24.447345 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:24.448243 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:24.454154 1574 catalog_manager.cc:1347] Generated new cluster ID: cc02910cb347401ab75bc32fe5defcc4
I20260929 14:47:24.454224 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:24.462925 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:24.464362 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:24.474622 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Generated new TSK 0
I20260929 14:47:24.475427 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:24.505021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33037
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:24.622378 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:24.622658 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:24.622704 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:24.626752 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:24.626818 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:24.626859 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:24.626891 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:24.632087 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:24.633136 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:24.634110 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:24.643656 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:24.643718 1580 server_base.cc:1034] running on GCE node
W20260929 14:47:24.643661 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:24.643662 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:24.644402 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:24.645057 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:24.646222 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790693244646206 us; error 27 us; skew 500 ppm
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:24.649118 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:24.650017 1580 webserver.cc:460] Webserver started at http://127.0.98.1:36123/ using document root <none> and password file <none>
I20260929 14:47:24.650606 1580 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:24.650700 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:24.650938 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:24.653100 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "2883d45fe22743888b19ca855c23bd92"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "d25083aeee5e3a7fc6ed21482b192889"
server_key_iv: "a93023fceae01d61fb69718a7cf5f31f"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.653671 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2883d45fe22743888b19ca855c23bd92"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "d25083aeee5e3a7fc6ed21482b192889"
server_key_iv: "a93023fceae01d61fb69718a7cf5f31f"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.657474 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:24.659930 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:24.661180 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:24.661312 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "2883d45fe22743888b19ca855c23bd92"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "d25083aeee5e3a7fc6ed21482b192889"
server_key_iv: "a93023fceae01d61fb69718a7cf5f31f"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.661427 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:24.713142 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:24.716085 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:24.716317 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:24.717761 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:24.717834 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:24.717905 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:24.717957 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:24.728981 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42799
I20260929 14:47:24.729153 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42799 every 8 connection(s)
I20260929 14:47:24.729904 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:24.731549 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.718699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35785 (local address 127.0.98.62:33037)
0929 14:47:24.718823 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:24.718827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:24.719870 (+ 1043us) server_negotiation.cc:407] Connection header received
0929 14:47:24.720967 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:24.720972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:24.721031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:24.721082 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:24.722361 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.722988 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.724337 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.724566 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.724682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.727773 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:24.727776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:24.727778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:24.727801 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:24.729406 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.729894 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.729899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.729901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.729981 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.730414 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.730417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.730419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.730669 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:24.730769 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:24.731261 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 14:47:24.731385 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:47:24.732344 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.719257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.1:35785)
0929 14:47:24.719720 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:24.719761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:24.720713 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:24.721385 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:24.721391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:24.721733 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:24.722211 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.722224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.723147 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.723150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.724213 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.724222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.724728 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.724732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.724896 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.725569 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:24.725584 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:24.727605 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:24.729525 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.729530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.729543 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.729790 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.730097 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.730100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.730102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.730280 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.730788 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:24.730794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:24.731023 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 14:47:24.731207 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 14:47:24.731507 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260929 14:47:24.733009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1580
I20260929 14:47:24.733160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:24.733436 392 external_mini_cluster.cc:1428] Setting key f87aa984c4741055ecc70b62013302a3
I20260929 14:47:24.733714 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33037
I20260929 14:47:24.733850 1711 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:24.734347 1711 heartbeater.cc:511] Master 127.0.98.62:33037 requested a full tablet report, sending...
I20260929 14:47:24.735906 1507 ts_manager.cc:194] Registered new tserver with Master: 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799)
I20260929 14:47:24.737498 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35785
I20260929 14:47:24.743597 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.735878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.1:43449)
0929 14:47:24.736252 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:24.736269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:24.736371 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:24.736610 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:24.736617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:24.736835 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:24.736936 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.736941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.737736 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.737739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.738452 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.738462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.739103 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.739106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.739191 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.739715 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:24.739729 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:24.740089 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:24.741954 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.741956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.741958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.742148 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.742542 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.742545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.742547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.742634 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.743044 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:24.743046 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:24.743138 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 14:47:24.743262 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 14:47:24.743392 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260929 14:47:24.743644 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.735820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43449 (local address 127.0.98.62:33037)
0929 14:47:24.735952 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:24.735958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:24.736347 (+ 389us) server_negotiation.cc:407] Connection header received
0929 14:47:24.736447 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:24.736450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:24.736488 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:24.736571 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:24.737050 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.737556 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.738570 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.738967 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.739099 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.740255 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:24.740260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:24.740265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:24.740296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:24.741856 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.742313 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.742318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.742321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.742410 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.742729 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.742733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.742734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.742939 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:24.743045 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:24.743359 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:47:24.743488 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:24.764520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33037
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:24.874081 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:24.874328 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:24.874372 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:24.878437 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:24.878492 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:24.878530 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:24.878561 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:24.883550 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:24.884552 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:24.885499 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:47:24.894708 1720 server_base.cc:1034] running on GCE node
W20260929 14:47:24.894695 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:24.894672 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:24.894672 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:24.895502 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:24.896107 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:24.897275 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790693244897260 us; error 28 us; skew 500 ppm
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:24.899995 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:24.900964 1720 webserver.cc:460] Webserver started at http://127.0.98.2:35297/ using document root <none> and password file <none>
I20260929 14:47:24.901525 1720 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:24.901588 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:24.901844 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:24.904032 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "801dbd8b9bd022e71b0c2f248ae8803b"
server_key_iv: "41c6bd9c6eb1ce142559118e9ee47138"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.904654 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "801dbd8b9bd022e71b0c2f248ae8803b"
server_key_iv: "41c6bd9c6eb1ce142559118e9ee47138"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.908226 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:24.910547 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:24.911795 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:24.911949 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d"
format_stamp: "Formatted at 2026-09-29 14:47:24 on dist-test-slave-5hd1"
server_key: "801dbd8b9bd022e71b0c2f248ae8803b"
server_key_iv: "41c6bd9c6eb1ce142559118e9ee47138"
server_key_version: "encryptionkey@0"
I20260929 14:47:24.912065 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:24.925877 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:24.929312 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:24.929510 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:24.930999 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:24.931082 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:24.931151 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:24.931180 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:24 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:24.941524 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34159
I20260929 14:47:24.941605 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34159 every 8 connection(s)
I20260929 14:47:24.942667 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:24.945848 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.931906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55863 (local address 127.0.98.62:33037)
0929 14:47:24.932030 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:24.932034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:24.933060 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 14:47:24.934278 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:24.934284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:24.934355 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:24.934460 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:24.935992 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.936913 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.938691 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.939054 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.939177 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.942115 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:24.942119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:24.942121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:24.942150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:24.943924 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.944472 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.944474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.944476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.944549 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.944955 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.944958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.944959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.945130 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:24.945204 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:24.945641 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 14:47:24.945733 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:24.946652 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.932453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.2:55863)
0929 14:47:24.932919 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:24.932959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:24.934015 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:24.934668 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:24.934679 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:24.935141 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:24.935809 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.935832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.937072 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.937076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.938562 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.938575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.939196 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.939199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.939332 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.940100 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:24.940119 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:24.941954 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:24.944034 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.944037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.944048 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.944380 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.944653 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.944655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.944657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.944844 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.945217 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:24.945222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:24.945429 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 14:47:24.945574 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 14:47:24.945810 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260929 14:47:24.947860 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33037
I20260929 14:47:24.948007 1851 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:24.948544 1851 heartbeater.cc:511] Master 127.0.98.62:33037 requested a full tablet report, sending...
I20260929 14:47:24.949580 1508 ts_manager.cc:194] Registered new tserver with Master: 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2:34159)
I20260929 14:47:24.950243 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55863
I20260929 14:47:24.950852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1720
I20260929 14:47:24.950977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:24.951249 392 external_mini_cluster.cc:1428] Setting key aa3797a1b1fa08cd3126050ea0c2aa11
I20260929 14:47:24.957860 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.950063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38223 (local address 127.0.98.62:33037)
0929 14:47:24.950186 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:24.950190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:24.950374 (+ 184us) server_negotiation.cc:407] Connection header received
0929 14:47:24.950538 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:24.950541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:24.950587 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:24.950665 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:24.951090 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.951786 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.952942 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.953149 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.953221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.954418 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:24.954420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:24.954423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:24.954442 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:24.956101 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.956621 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.956623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.956624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.956694 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:24.956989 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:24.956991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:24.956992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:24.957185 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:24.957260 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:24.957622 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 14:47:24.957756 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:47:24.957960 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:24.949994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.2:38223)
0929 14:47:24.950295 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:24.950311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:24.950434 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:24.950681 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:24.950684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:24.950867 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:24.950981 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.950988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.951937 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.951940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.952818 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:24.952827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:24.953256 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:24.953259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:24.953346 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:24.953786 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:24.953805 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:24.954288 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:24.956254 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.956260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.956265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.956519 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.956782 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:24.956788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:24.956793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:24.956891 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:24.957282 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:24.957288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:24.957401 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 14:47:24.957525 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 14:47:24.957677 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":22,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:24.979946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33037
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:25.093557 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:25.093881 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:25.093955 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:25.098173 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:25.098271 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:25.098341 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:25.098429 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:25.104002 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:25.105159 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:25.106211 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:25.116417 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:25.116473 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:25.116699 1860 server_base.cc:1034] running on GCE node
W20260929 14:47:25.116425 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:25.117318 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:25.118002 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:25.119211 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790693245119196 us; error 31 us; skew 500 ppm
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:25.122131 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:25.123059 1860 webserver.cc:460] Webserver started at http://127.0.98.3:34511/ using document root <none> and password file <none>
I20260929 14:47:25.123651 1860 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:25.123739 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:25.123962 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:25.126199 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "84e1ed876b514410864d947cc44c25e6"
format_stamp: "Formatted at 2026-09-29 14:47:25 on dist-test-slave-5hd1"
server_key: "ac0a2f5bbfd03e67f75a123ad8599d8c"
server_key_iv: "69cbf569225cd6602a06ad0586125751"
server_key_version: "encryptionkey@0"
I20260929 14:47:25.126765 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "84e1ed876b514410864d947cc44c25e6"
format_stamp: "Formatted at 2026-09-29 14:47:25 on dist-test-slave-5hd1"
server_key: "ac0a2f5bbfd03e67f75a123ad8599d8c"
server_key_iv: "69cbf569225cd6602a06ad0586125751"
server_key_version: "encryptionkey@0"
I20260929 14:47:25.130532 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:47:25.133057 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:25.134596 1860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:25.134693 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "84e1ed876b514410864d947cc44c25e6"
format_stamp: "Formatted at 2026-09-29 14:47:25 on dist-test-slave-5hd1"
server_key: "ac0a2f5bbfd03e67f75a123ad8599d8c"
server_key_iv: "69cbf569225cd6602a06ad0586125751"
server_key_version: "encryptionkey@0"
I20260929 14:47:25.134780 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:25.157235 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:25.160272 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:25.160431 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:25.161788 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:25.161836 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:25.161901 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:25.161933 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:25.172856 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42693
I20260929 14:47:25.172890 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42693 every 8 connection(s)
I20260929 14:47:25.173959 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:25.176242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 1860
I20260929 14:47:25.176393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:25.176761 392 external_mini_cluster.cc:1428] Setting key 8620057195fa144ddd703810f273b7a6
I20260929 14:47:25.177629 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.162737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43057 (local address 127.0.98.62:33037)
0929 14:47:25.162866 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.162869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.163966 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 14:47:25.165169 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.165173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.165234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.165347 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.166984 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.167884 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.169771 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.170106 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.170250 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.173312 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.173317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.173320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.173351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.175061 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.175781 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.175784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.175785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.175866 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.176276 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.176279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.176281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.176543 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.176759 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.177328 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.177461 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:25.178587 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.163320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.3:43057)
0929 14:47:25.163816 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.163858 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.164909 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.165581 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.165592 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.166132 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:25.166811 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.166830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.168030 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.168035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.169633 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.169644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.170290 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.170293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.170441 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.171185 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:25.171210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:25.173137 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:25.175186 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.175191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.175202 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.175656 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.175966 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.175968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.175970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.176113 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.176741 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:25.176748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:25.177067 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.177269 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.177617 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":163,"threads_started":1}
I20260929 14:47:25.179947 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33037
I20260929 14:47:25.180131 1991 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:25.180794 1991 heartbeater.cc:511] Master 127.0.98.62:33037 requested a full tablet report, sending...
I20260929 14:47:25.181902 1507 ts_manager.cc:194] Registered new tserver with Master: 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.182606 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43057
I20260929 14:47:25.189778 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.182180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49917 (local address 127.0.98.62:33037)
0929 14:47:25.182309 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.182312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.183068 (+ 756us) server_negotiation.cc:407] Connection header received
0929 14:47:25.183189 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.183192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.183233 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.183359 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.183799 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.184435 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.185235 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.185472 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.185590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.186716 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.186720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.186722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.186748 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.188045 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.188515 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.188518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.188520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.188606 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.188886 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.188891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.188893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.189088 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.189196 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.189459 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.189581 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:47:25.189785 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.182081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.98.3:49917)
0929 14:47:25.182937 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.182949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.183103 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.183353 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.183357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.183583 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:25.183684 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.183690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.184540 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.184544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.185116 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.185124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.185585 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.185589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.185681 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.186218 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:25.186229 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:25.186580 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:25.188166 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.188170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.188197 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.188405 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.188706 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.188709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.188711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.188796 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.189183 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:25.189186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:25.189288 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.189394 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.189579 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":114,"threads_started":1}
I20260929 14:47:25.191411 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:25.230510 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.220076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.98.62:33037)
0929 14:47:25.220287 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.220292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.220313 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:47:25.220400 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.220404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.220489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.220660 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.221813 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.222768 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.223890 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.224154 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.224326 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.226949 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.226952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.226954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.226983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.228466 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.229230 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.229233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.229234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.229318 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.229658 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.229661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.229662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.229860 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.229976 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.230269 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.230376 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:47:25.234202 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:25.237329 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:25.252399 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.246830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.0.98.1:42799)
0929 14:47:25.247375 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.247383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.247413 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:47:25.247506 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.247511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.247741 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.247896 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.248212 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.249381 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.250963 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.251622 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.251749 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.251911 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.252122 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.252250 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":81,"threads_started":1}
I20260929 14:47:25.252399 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.246199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.0.1:52060)
0929 14:47:25.247011 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.247065 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.247256 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.247910 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.247918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.247943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.248076 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.248083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.249509 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.249512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.250821 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.250833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.251745 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.251749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.251877 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.251937 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.252087 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.252243 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":4,"spinlock_wait_cycles":3968,"thread_start_us":60,"threads_started":1}
I20260929 14:47:25.252614 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.246407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34159 (local address 127.0.0.1:38768)
0929 14:47:25.247112 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.247126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.247256 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.248331 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.248334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.248352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.248430 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.248436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.249770 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.249774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.251084 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.251093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.252225 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.252228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.252326 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.252350 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.252400 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.252467 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":85,"spinlock_wait_cycles":9856,"thread_start_us":137,"threads_started":1}
I20260929 14:47:25.253293 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.246577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.0.98.2:34159)
0929 14:47:25.247863 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.247875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.247896 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:47:25.247963 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.247968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.248208 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.248356 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.248567 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.249619 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.251205 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.252061 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.252815 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.252983 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.253034 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.253140 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":117,"threads_started":1}
I20260929 14:47:25.253643 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.246880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42693 (local address 127.0.0.1:47098)
0929 14:47:25.248538 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.248549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.248700 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.249108 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.249110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.249128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.249211 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.249219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.250669 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.250672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.251917 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.251928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.253242 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.253246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.253345 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.253370 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.253448 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.253513 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":68,"thread_start_us":97,"threads_started":1}
I20260929 14:47:25.254021 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.247295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.0.98.3:42693)
0929 14:47:25.247620 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.247630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.248659 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 14:47:25.248836 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.248840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.248975 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.249127 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.249334 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.250523 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.252062 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.252930 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.253034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.253182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.253790 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.253883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260929 14:47:25.255926 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 (DEFAULT_TABLE table=test-table [id=077c3b4358c04ada9e204f0fed10c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.255923 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 (DEFAULT_TABLE table=test-table [id=077c3b4358c04ada9e204f0fed10c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.256999 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceff59c5c47417882b08bc59c5d6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.257014 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceff59c5c47417882b08bc59c5d6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.259065 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 (DEFAULT_TABLE table=test-table [id=077c3b4358c04ada9e204f0fed10c299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.260210 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceff59c5c47417882b08bc59c5d6df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.264155 2014 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Bootstrap starting.
I20260929 14:47:25.266584 2014 tablet_bootstrap.cc:654] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.267309 2014 log.cc:824] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:25.267869 2016 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Bootstrap starting.
I20260929 14:47:25.268030 2015 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Bootstrap starting.
I20260929 14:47:25.269670 2014 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: No bootstrap required, opened a new log
I20260929 14:47:25.269868 2014 ts_tablet_manager.cc:1397] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 14:47:25.270058 2016 tablet_bootstrap.cc:654] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.270418 2015 tablet_bootstrap.cc:654] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.270829 2016 log.cc:824] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:25.271229 2015 log.cc:824] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:25.272778 2016 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: No bootstrap required, opened a new log
I20260929 14:47:25.272971 2016 ts_tablet_manager.cc:1397] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:47:25.273743 2014 raft_consensus.cc:348] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.273921 2015 tablet_bootstrap.cc:492] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: No bootstrap required, opened a new log
I20260929 14:47:25.273998 2014 raft_consensus.cc:374] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.274049 2014 raft_consensus.cc:729] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827f714b0e7a4fcb96aa5fa06abb9b1d, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.274118 2015 ts_tablet_manager.cc:1397] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:47:25.274519 2014 consensus_queue.cc:260] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.275225 1851 heartbeater.cc:503] Master 127.0.98.62:33037 was elected leader, sending a full tablet report...
I20260929 14:47:25.275610 2014 ts_tablet_manager.cc:1428] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 14:47:25.276953 2016 raft_consensus.cc:348] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.277220 2016 raft_consensus.cc:374] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.277278 2016 raft_consensus.cc:729] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e1ed876b514410864d947cc44c25e6, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.277719 2016 consensus_queue.cc:260] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.278504 1991 heartbeater.cc:503] Master 127.0.98.62:33037 was elected leader, sending a full tablet report...
I20260929 14:47:25.278769 2016 ts_tablet_manager.cc:1428] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:25.279989 2015 raft_consensus.cc:348] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.280296 2015 raft_consensus.cc:374] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.280364 2015 raft_consensus.cc:729] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2883d45fe22743888b19ca855c23bd92, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.280797 2015 consensus_queue.cc:260] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.281418 1711 heartbeater.cc:503] Master 127.0.98.62:33037 was elected leader, sending a full tablet report...
I20260929 14:47:25.281774 2015 ts_tablet_manager.cc:1428] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260929 14:47:25.425009 1992 tablet.cc:2369] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:47:25.444262 1852 tablet.cc:2369] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:25.463258 2020 raft_consensus.cc:482] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:25.463491 2020 raft_consensus.cc:504] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.464803 2020 leader_election.cc:288] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.469053 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.465189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47775 (local address 127.0.98.1:42799)
0929 14:47:25.465357 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.465364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.465448 (+ 84us) server_negotiation.cc:407] Connection header received
0929 14:47:25.465653 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.465657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.465716 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.465803 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.466196 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.466805 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.467947 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.468454 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.468571 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.468638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.468821 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.468900 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:47:25.469103 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.465059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.98.2:47775)
0929 14:47:25.465327 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.465341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.465495 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.465843 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.465846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.465933 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.466082 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.466090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.466919 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.466922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.467808 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.467818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.468581 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.468585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.468682 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.468712 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.468783 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.468888 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"spinlock_wait_cycles":105728,"thread_start_us":78,"threads_started":1}
I20260929 14:47:25.469074 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.465602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49845 (local address 127.0.98.3:42693)
0929 14:47:25.465727 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.465730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.465845 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:47:25.466032 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.466035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.466087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.466172 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.466460 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.467006 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.467992 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.468466 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.468590 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.468651 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.468858 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.468948 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:47:25.469122 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.465502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42693 (local address 127.0.98.2:49845)
0929 14:47:25.465760 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.465774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.465916 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.466196 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.466200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.466217 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.466323 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.466330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.467126 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.467128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.467836 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.467844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.468604 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.468607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.468716 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.468733 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.468831 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.468900 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"spinlock_wait_cycles":8448,"thread_start_us":56,"threads_started":1}
I20260929 14:47:25.469858 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceff59c5c47417882b08bc59c5d6df2" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6" is_pre_election: true
I20260929 14:47:25.469854 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceff59c5c47417882b08bc59c5d6df2" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883d45fe22743888b19ca855c23bd92" is_pre_election: true
I20260929 14:47:25.470170 1665 raft_consensus.cc:2454] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 0.
I20260929 14:47:25.470172 1945 raft_consensus.cc:2454] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 0.
I20260929 14:47:25.470660 1737 leader_election.cc:302] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2883d45fe22743888b19ca855c23bd92, 827f714b0e7a4fcb96aa5fa06abb9b1d; no voters:
I20260929 14:47:25.470942 2020 raft_consensus.cc:2781] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:25.471031 2020 raft_consensus.cc:482] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:25.471076 2020 raft_consensus.cc:3037] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.472134 2020 raft_consensus.cc:504] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.472553 2020 leader_election.cc:288] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 election: Requested vote from peers 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.472934 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceff59c5c47417882b08bc59c5d6df2" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883d45fe22743888b19ca855c23bd92"
I20260929 14:47:25.472980 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceff59c5c47417882b08bc59c5d6df2" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6"
I20260929 14:47:25.473078 1665 raft_consensus.cc:3037] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.473104 1945 raft_consensus.cc:3037] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.474155 1665 raft_consensus.cc:2454] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 1.
I20260929 14:47:25.474519 1737 leader_election.cc:302] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2883d45fe22743888b19ca855c23bd92, 827f714b0e7a4fcb96aa5fa06abb9b1d; no voters:
I20260929 14:47:25.474568 1945 raft_consensus.cc:2454] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 1.
I20260929 14:47:25.474864 2020 raft_consensus.cc:2781] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:25.475085 2020 raft_consensus.cc:686] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 LEADER]: Becoming Leader. State: Replica: 827f714b0e7a4fcb96aa5fa06abb9b1d, State: Running, Role: LEADER
I20260929 14:47:25.475456 2020 consensus_queue.cc:237] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.478775 1507 catalog_manager.cc:5495] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } health_report { overall_health: UNKNOWN } } }
W20260929 14:47:25.481236 1712 tablet.cc:2369] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:25.550655 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.546838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.0.98.2:34159)
0929 14:47:25.546963 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.546967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.546980 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:25.547173 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.547175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.547234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.547326 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:25.547573 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.548190 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.549068 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.549295 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.549378 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.549776 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.550278 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.550367 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.550478 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.550540 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:47:25.556414 1665 raft_consensus.cc:1261] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Refusing update from remote peer 827f714b0e7a4fcb96aa5fa06abb9b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.556414 1945 raft_consensus.cc:1261] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Refusing update from remote peer 827f714b0e7a4fcb96aa5fa06abb9b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.557173 2020 consensus_queue.cc:1035] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.557443 2025 consensus_queue.cc:1035] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.565991 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334679533791969280 to 7334679533470728192. Current Snapshot: MvccSnapshot[applied={T|T < 7334679533791969280}]
I20260929 14:47:25.567114 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334679533791969280 to 7334679533470728192. Current Snapshot: MvccSnapshot[applied={T|T < 7334679533791969280}]
I20260929 14:47:25.576483 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:25.576680 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:25.579082 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Sending DeleteTablet for 3 replicas of tablet 4ceff59c5c47417882b08bc59c5d6df2
I20260929 14:47:25.579866 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52060
I20260929 14:47:25.579866 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260929 14:47:25.579919 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ceff59c5c47417882b08bc59c5d6df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47098
I20260929 14:47:25.580394 2038 tablet_replica.cc:330] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: stopping tablet replica
I20260929 14:47:25.580415 2039 tablet_replica.cc:330] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: stopping tablet replica
I20260929 14:47:25.580760 2039 raft_consensus.cc:2229] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:25.580713 2038 raft_consensus.cc:2229] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:25.580999 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:47:25.581094 2038 raft_consensus.cc:2258] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.581131 2037 tablet_replica.cc:330] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: stopping tablet replica
W20260929 14:47:25.581440 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:25.581882 2037 raft_consensus.cc:2229] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:25.582299 2037 raft_consensus.cc:2258] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.582728 2039 raft_consensus.cc:2258] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.582924 2038 ts_tablet_manager.cc:1905] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.584033 2037 ts_tablet_manager.cc:1905] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.585844 2038 ts_tablet_manager.cc:1918] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:25.585996 2038 log.cc:1196] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/4ceff59c5c47417882b08bc59c5d6df2
I20260929 14:47:25.586372 2038 ts_tablet_manager.cc:1939] T 4ceff59c5c47417882b08bc59c5d6df2 P 2883d45fe22743888b19ca855c23bd92: Deleting consensus metadata
I20260929 14:47:25.586802 2037 ts_tablet_manager.cc:1918] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:25.586897 2037 log.cc:1196] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/4ceff59c5c47417882b08bc59c5d6df2
I20260929 14:47:25.587229 2037 ts_tablet_manager.cc:1939] T 4ceff59c5c47417882b08bc59c5d6df2 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting consensus metadata
I20260929 14:47:25.587554 1785 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 (DEFAULT_TABLE table=test-table [id=2e4f389a367f436190cbe4ef595ca931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.587808 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5aab7c68644ff98276dff13ee2a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.588798 1495 catalog_manager.cc:4841] TS 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2:34159): tablet 4ceff59c5c47417882b08bc59c5d6df2 (table test-table [id=077c3b4358c04ada9e204f0fed10c299]) successfully deleted
I20260929 14:47:25.588872 2039 ts_tablet_manager.cc:1905] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.589285 1493 catalog_manager.cc:4841] TS 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799): tablet 4ceff59c5c47417882b08bc59c5d6df2 (table test-table [id=077c3b4358c04ada9e204f0fed10c299]) successfully deleted
I20260929 14:47:25.589500 1645 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 (DEFAULT_TABLE table=test-table [id=2e4f389a367f436190cbe4ef595ca931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.589740 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5aab7c68644ff98276dff13ee2a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.590077 2014 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Bootstrap starting.
I20260929 14:47:25.590435 1925 tablet_service.cc:1467] Processing CreateTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 (DEFAULT_TABLE table=test-table [id=2e4f389a367f436190cbe4ef595ca931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:25.591095 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa5aab7c68644ff98276dff13ee2a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.591472 2014 tablet_bootstrap.cc:654] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.591962 2039 ts_tablet_manager.cc:1918] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:25.592111 2039 log.cc:1196] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/4ceff59c5c47417882b08bc59c5d6df2
I20260929 14:47:25.592514 2039 ts_tablet_manager.cc:1939] T 4ceff59c5c47417882b08bc59c5d6df2 P 84e1ed876b514410864d947cc44c25e6: Deleting consensus metadata
I20260929 14:47:25.592628 2014 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: No bootstrap required, opened a new log
I20260929 14:47:25.592712 2014 ts_tablet_manager.cc:1397] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 14:47:25.593149 2014 raft_consensus.cc:348] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.593268 2014 raft_consensus.cc:374] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.593309 2014 raft_consensus.cc:729] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827f714b0e7a4fcb96aa5fa06abb9b1d, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.593428 2014 consensus_queue.cc:260] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.593490 2015 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Bootstrap starting.
I20260929 14:47:25.593688 2014 ts_tablet_manager.cc:1428] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 14:47:25.593899 2016 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Bootstrap starting.
I20260929 14:47:25.594377 2015 tablet_bootstrap.cc:654] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.594796 2016 tablet_bootstrap.cc:654] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.595060 1493 catalog_manager.cc:4841] TS 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693): tablet 4ceff59c5c47417882b08bc59c5d6df2 (table test-table [id=077c3b4358c04ada9e204f0fed10c299]) successfully deleted
I20260929 14:47:25.596242 2015 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: No bootstrap required, opened a new log
I20260929 14:47:25.596321 2015 ts_tablet_manager.cc:1397] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:25.597009 2016 tablet_bootstrap.cc:492] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: No bootstrap required, opened a new log
I20260929 14:47:25.597088 2016 ts_tablet_manager.cc:1397] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:25.597118 2015 raft_consensus.cc:348] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.597220 2015 raft_consensus.cc:374] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.597250 2015 raft_consensus.cc:729] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2883d45fe22743888b19ca855c23bd92, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.597360 2015 consensus_queue.cc:260] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.597507 2016 raft_consensus.cc:348] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.597611 2015 ts_tablet_manager.cc:1428] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:25.597637 2016 raft_consensus.cc:374] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.597671 2016 raft_consensus.cc:729] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e1ed876b514410864d947cc44c25e6, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.597813 2016 consensus_queue.cc:260] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.598055 2016 ts_tablet_manager.cc:1428] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:47:25.624475 2022 raft_consensus.cc:482] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:25.624632 2022 raft_consensus.cc:504] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.625859 2022 leader_election.cc:288] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2:34159), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.630298 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.626467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39375 (local address 127.0.98.3:42693)
0929 14:47:25.626574 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.626579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.626593 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:25.626674 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.626677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.626722 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.626799 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.627135 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.627920 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.629310 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.629714 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.629845 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.629913 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.630113 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.630186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:47:25.630314 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.626100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42693 (local address 127.0.98.1:39375)
0929 14:47:25.626481 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.626492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.626563 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.626831 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.626834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.626854 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.627001 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.627009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.628030 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.628033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.629196 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.629205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.629856 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.629859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.629967 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.629993 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.630094 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.630167 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20260929 14:47:25.630503 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.626235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53383 (local address 127.0.98.2:34159)
0929 14:47:25.626350 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.626356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.626427 (+ 71us) server_negotiation.cc:407] Connection header received
0929 14:47:25.626670 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.626677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.626724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.626795 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:25.627183 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.627675 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.629237 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.629852 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.629951 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.630010 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.630316 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.630384 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:47:25.630869 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.626036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34159 (local address 127.0.98.1:53383)
0929 14:47:25.626349 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.626364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.626503 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.626928 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.626930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.626945 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:25.627090 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.627097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.627791 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.627796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.629106 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.629116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.630078 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.630082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.630184 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.630206 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.630640 (+ 434us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.630710 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":69,"thread_start_us":80,"threads_started":1}
I20260929 14:47:25.631138 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5aab7c68644ff98276dff13ee2a90" candidate_uuid: "2883d45fe22743888b19ca855c23bd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6" is_pre_election: true
I20260929 14:47:25.631286 1945 raft_consensus.cc:2454] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2883d45fe22743888b19ca855c23bd92 in term 0.
I20260929 14:47:25.631433 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5aab7c68644ff98276dff13ee2a90" candidate_uuid: "2883d45fe22743888b19ca855c23bd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" is_pre_election: true
I20260929 14:47:25.631680 1805 raft_consensus.cc:2454] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2883d45fe22743888b19ca855c23bd92 in term 0.
I20260929 14:47:25.632086 1599 leader_election.cc:302] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2883d45fe22743888b19ca855c23bd92, 827f714b0e7a4fcb96aa5fa06abb9b1d; no voters:
I20260929 14:47:25.632380 2022 raft_consensus.cc:2781] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:25.632440 2022 raft_consensus.cc:482] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:25.632483 2022 raft_consensus.cc:3037] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.633791 2022 raft_consensus.cc:504] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.634131 2022 leader_election.cc:288] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [CANDIDATE]: Term 1 election: Requested vote from peers 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2:34159), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.634408 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5aab7c68644ff98276dff13ee2a90" candidate_uuid: "2883d45fe22743888b19ca855c23bd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d"
I20260929 14:47:25.634549 1805 raft_consensus.cc:3037] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.635557 1805 raft_consensus.cc:2454] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2883d45fe22743888b19ca855c23bd92 in term 1.
I20260929 14:47:25.635823 1599 leader_election.cc:302] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2883d45fe22743888b19ca855c23bd92, 827f714b0e7a4fcb96aa5fa06abb9b1d; no voters:
I20260929 14:47:25.635936 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa5aab7c68644ff98276dff13ee2a90" candidate_uuid: "2883d45fe22743888b19ca855c23bd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6"
I20260929 14:47:25.635990 2022 raft_consensus.cc:2781] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:25.636066 1945 raft_consensus.cc:3037] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.636261 2022 raft_consensus.cc:686] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 1 LEADER]: Becoming Leader. State: Replica: 2883d45fe22743888b19ca855c23bd92, State: Running, Role: LEADER
I20260929 14:47:25.636535 2022 consensus_queue.cc:237] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.637133 1945 raft_consensus.cc:2454] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2883d45fe22743888b19ca855c23bd92 in term 1.
I20260929 14:47:25.638924 1508 catalog_manager.cc:5495] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2883d45fe22743888b19ca855c23bd92 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2883d45fe22743888b19ca855c23bd92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:25.679363 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.670109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.98.62:33037)
0929 14:47:25.670249 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.670253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.670453 (+ 200us) server_negotiation.cc:407] Connection header received
0929 14:47:25.670577 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.670579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.670619 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.670687 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.671519 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.672038 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.673193 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.673406 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.673511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.676083 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.676086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.676088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.676107 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.677644 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.678131 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.678135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.678136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.678209 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.678475 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.678480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.678482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.678657 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.678757 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.679056 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.679219 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":57}
I20260929 14:47:25.679365 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.670040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33037 (local address 127.0.0.1:35040)
0929 14:47:25.670356 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.670371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.670489 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.670748 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.670752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.671285 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:25.671405 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.671413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.672145 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.672149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.673072 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.673083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.673540 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.673545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.673632 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.674231 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:25.674245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:25.675927 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:25.677771 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.677781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.677786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.678022 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.678297 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:25.678301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:25.678303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:25.678386 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:25.678766 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:25.678774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:25.678883 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.679001 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.679163 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260929 14:47:25.683362 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:47:25.684084 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:25.689076 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 693fca9b06544d08a360cdf83a9fe972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d66bb2195542b7a593e4d75dcae968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:25.689499 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693fca9b06544d08a360cdf83a9fe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.689672 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 693fca9b06544d08a360cdf83a9fe972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d66bb2195542b7a593e4d75dcae968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:25.690071 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693fca9b06544d08a360cdf83a9fe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.692874 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 693fca9b06544d08a360cdf83a9fe972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88d66bb2195542b7a593e4d75dcae968]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:25.693090 2015 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: Bootstrap starting.
I20260929 14:47:25.693279 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693fca9b06544d08a360cdf83a9fe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:25.693513 2014 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Bootstrap starting.
I20260929 14:47:25.694038 2015 tablet_bootstrap.cc:654] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.694543 2014 tablet_bootstrap.cc:654] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.695609 2015 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: No bootstrap required, opened a new log
I20260929 14:47:25.695683 2015 ts_tablet_manager.cc:1397] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:25.696141 2015 raft_consensus.cc:348] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.696282 2015 raft_consensus.cc:374] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.696314 2015 raft_consensus.cc:729] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2883d45fe22743888b19ca855c23bd92, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.696439 2015 consensus_queue.cc:260] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.696704 2015 ts_tablet_manager.cc:1428] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:25.696954 2016 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: Bootstrap starting.
I20260929 14:47:25.696908 2045 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.697059 2045 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.697959 2014 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: No bootstrap required, opened a new log
I20260929 14:47:25.698015 2016 tablet_bootstrap.cc:654] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:25.698050 2014 ts_tablet_manager.cc:1397] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260929 14:47:25.698698 2014 raft_consensus.cc:348] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.699002 2014 raft_consensus.cc:374] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.699056 2014 raft_consensus.cc:729] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827f714b0e7a4fcb96aa5fa06abb9b1d, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.699208 2014 consensus_queue.cc:260] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.699467 2020 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.699632 2020 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.699680 2016 tablet_bootstrap.cc:492] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: No bootstrap required, opened a new log
I20260929 14:47:25.699776 2016 ts_tablet_manager.cc:1397] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:25.700263 2016 raft_consensus.cc:348] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.700310 2014 ts_tablet_manager.cc:1428] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:47:25.700374 2016 raft_consensus.cc:374] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:25.700410 2016 raft_consensus.cc:729] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e1ed876b514410864d947cc44c25e6, State: Initialized, Role: FOLLOWER
I20260929 14:47:25.700531 2016 consensus_queue.cc:260] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.700802 2016 ts_tablet_manager.cc:1428] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:25.700805 2021 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.701388 2021 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.706507 2020 raft_consensus.cc:482] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:25.706619 2020 raft_consensus.cc:504] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.707062 2020 leader_election.cc:288] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.707453 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693fca9b06544d08a360cdf83a9fe972" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6" is_pre_election: true
I20260929 14:47:25.707453 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693fca9b06544d08a360cdf83a9fe972" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883d45fe22743888b19ca855c23bd92" is_pre_election: true
I20260929 14:47:25.707597 1665 raft_consensus.cc:2454] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 0.
I20260929 14:47:25.707597 1945 raft_consensus.cc:2454] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 0.
I20260929 14:47:25.707878 1737 leader_election.cc:302] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827f714b0e7a4fcb96aa5fa06abb9b1d, 84e1ed876b514410864d947cc44c25e6; no voters:
I20260929 14:47:25.708050 2020 raft_consensus.cc:2781] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:25.708112 2020 raft_consensus.cc:482] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:25.708163 2020 raft_consensus.cc:3037] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.709708 2020 raft_consensus.cc:504] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.710104 2020 leader_election.cc:288] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 election: Requested vote from peers 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799), 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693)
I20260929 14:47:25.710484 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693fca9b06544d08a360cdf83a9fe972" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e1ed876b514410864d947cc44c25e6"
I20260929 14:47:25.710508 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693fca9b06544d08a360cdf83a9fe972" candidate_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2883d45fe22743888b19ca855c23bd92"
I20260929 14:47:25.710587 1945 raft_consensus.cc:3037] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.710639 1665 raft_consensus.cc:3037] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:25.711462 1945 raft_consensus.cc:2454] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 1.
I20260929 14:47:25.711781 1737 leader_election.cc:302] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827f714b0e7a4fcb96aa5fa06abb9b1d, 84e1ed876b514410864d947cc44c25e6; no voters:
I20260929 14:47:25.711797 1665 raft_consensus.cc:2454] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f714b0e7a4fcb96aa5fa06abb9b1d in term 1.
I20260929 14:47:25.711934 2020 raft_consensus.cc:2781] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:25.712041 2020 raft_consensus.cc:686] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 LEADER]: Becoming Leader. State: Replica: 827f714b0e7a4fcb96aa5fa06abb9b1d, State: Running, Role: LEADER
I20260929 14:47:25.712225 2020 consensus_queue.cc:237] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } }
I20260929 14:47:25.713240 2025 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: TxnStatusTablet state changed. Reason: New leader 827f714b0e7a4fcb96aa5fa06abb9b1d. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.713380 2025 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:25.714109 1508 catalog_manager.cc:5495] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:25.714808 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:47:25.714977 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:47:25.725982 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.722386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34159 (local address 127.0.0.1:38792)
0929 14:47:25.722528 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.722545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.722654 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.722955 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.722958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.722967 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:25.723068 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.723073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.724097 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.724100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.724764 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.724774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.725345 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.725348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.725410 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.725453 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.725742 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.725753 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.725802 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.725871 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:47:25.726012 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.722325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.0.98.2:34159)
0929 14:47:25.722479 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.722484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.722627 (+ 143us) server_negotiation.cc:407] Connection header received
0929 14:47:25.722764 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.722767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.722820 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.722913 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:25.723199 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.723963 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.724887 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.725223 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.725334 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.725574 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.725653 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.725732 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.725815 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.725867 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:47:25.728701 1665 raft_consensus.cc:1261] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Refusing update from remote peer 827f714b0e7a4fcb96aa5fa06abb9b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.728744 1945 raft_consensus.cc:1261] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Refusing update from remote peer 827f714b0e7a4fcb96aa5fa06abb9b1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.729224 2020 consensus_queue.cc:1035] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.729408 2025 consensus_queue.cc:1035] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.730801 2021 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: TxnStatusTablet state changed. Reason: New leader 827f714b0e7a4fcb96aa5fa06abb9b1d. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.730818 2045 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: TxnStatusTablet state changed. Reason: New leader 827f714b0e7a4fcb96aa5fa06abb9b1d. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.730918 2021 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.730926 2045 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.732874 2020 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.732978 2020 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:25.735339 2021 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.735447 2021 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 84e1ed876b514410864d947cc44c25e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.742384 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.737736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.0.98.1:42799)
0929 14:47:25.737843 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.737848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.738106 (+ 258us) server_negotiation.cc:407] Connection header received
0929 14:47:25.738180 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.738184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.738245 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.738347 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:25.738626 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.739447 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.740629 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.740975 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.741077 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.741401 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.741943 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.742068 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.742139 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.742215 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:47:25.746563 2025 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.746717 2025 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 827f714b0e7a4fcb96aa5fa06abb9b1d: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:25.748083 2045 tablet_replica.cc:439] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2883d45fe22743888b19ca855c23bd92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42799 } } peers { permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 } } peers { permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 } } }
I20260929 14:47:25.748216 2045 tablet_replica.cc:442] T 693fca9b06544d08a360cdf83a9fe972 P 2883d45fe22743888b19ca855c23bd92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:25.756498 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.752888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34159 (local address 127.0.98.1:52619)
0929 14:47:25.753214 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.753230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.753337 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.753611 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.753613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.753619 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:25.753702 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.753706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.754690 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.754693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.755298 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.755307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.755847 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.755850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.755908 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.755951 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.756252 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.756256 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.756308 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.756373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260929 14:47:25.756551 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.753269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52619 (local address 127.0.98.2:34159)
0929 14:47:25.753400 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.753405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.753418 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:25.753464 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.753467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.753516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.753593 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:25.753793 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.754583 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.755405 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.755748 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.755842 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.756075 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.756156 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.756243 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.756322 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.756391 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:47:25.763696 1945 raft_consensus.cc:1261] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Refusing update from remote peer 2883d45fe22743888b19ca855c23bd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.763863 1805 raft_consensus.cc:1261] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Refusing update from remote peer 2883d45fe22743888b19ca855c23bd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:25.764235 2045 consensus_queue.cc:1035] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e1ed876b514410864d947cc44c25e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.764415 2022 consensus_queue.cc:1035] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827f714b0e7a4fcb96aa5fa06abb9b1d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:25.788318 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.784393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42799 (local address 127.0.98.2:58477)
0929 14:47:25.784738 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:25.784754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:25.784856 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:25.785107 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:25.785110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:25.785118 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:25.785224 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.785229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.786163 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.786167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.787131 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:25.787141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.787557 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.787561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:25.787670 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.787720 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.788020 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.788029 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 14:47:25.788089 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:47:25.788190 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260929 14:47:25.788317 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.784492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58477 (local address 127.0.98.1:42799)
0929 14:47:25.784611 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.784616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.784813 (+ 197us) server_negotiation.cc:407] Connection header received
0929 14:47:25.784950 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.784953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.785012 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.785089 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:25.785335 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.786055 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.787244 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.787449 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.787532 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.787857 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:25.787942 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:25.788012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.788108 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.788157 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:25.810473 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:25.810642 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:25.812755 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1408951d9077499794309cc488320a0a: Sending DeleteTablet for 3 replicas of tablet dfa5aab7c68644ff98276dff13ee2a90
I20260929 14:47:25.813325 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38768
I20260929 14:47:25.813504 2037 tablet_replica.cc:330] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: stopping tablet replica
I20260929 14:47:25.813588 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52060
I20260929 14:47:25.813621 2037 raft_consensus.cc:2229] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:25.813731 2037 raft_consensus.cc:2258] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.813759 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa5aab7c68644ff98276dff13ee2a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47098
I20260929 14:47:25.813917 2038 tablet_replica.cc:330] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: stopping tablet replica
I20260929 14:47:25.814006 2038 raft_consensus.cc:2229] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:25.814268 2038 raft_consensus.cc:2258] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.814433 2039 tablet_replica.cc:330] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: stopping tablet replica
I20260929 14:47:25.814464 2037 ts_tablet_manager.cc:1905] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.814541 2039 raft_consensus.cc:2229] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:25.814646 2039 raft_consensus.cc:2258] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:25.815364 2038 ts_tablet_manager.cc:1905] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.817498 2037 ts_tablet_manager.cc:1918] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:25.817564 2037 log.cc:1196] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/dfa5aab7c68644ff98276dff13ee2a90
I20260929 14:47:25.817816 2037 ts_tablet_manager.cc:1939] T dfa5aab7c68644ff98276dff13ee2a90 P 827f714b0e7a4fcb96aa5fa06abb9b1d: Deleting consensus metadata
I20260929 14:47:25.818123 2038 ts_tablet_manager.cc:1918] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:25.818187 2038 log.cc:1196] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/dfa5aab7c68644ff98276dff13ee2a90
I20260929 14:47:25.818420 2038 ts_tablet_manager.cc:1939] T dfa5aab7c68644ff98276dff13ee2a90 P 2883d45fe22743888b19ca855c23bd92: Deleting consensus metadata
I20260929 14:47:25.819037 1493 catalog_manager.cc:4841] TS 2883d45fe22743888b19ca855c23bd92 (127.0.98.1:42799): tablet dfa5aab7c68644ff98276dff13ee2a90 (table test-table [id=2e4f389a367f436190cbe4ef595ca931]) successfully deleted
I20260929 14:47:25.819604 2039 ts_tablet_manager.cc:1905] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:25.822290 2039 ts_tablet_manager.cc:1918] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:25.822356 2039 log.cc:1196] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/dfa5aab7c68644ff98276dff13ee2a90
I20260929 14:47:25.822602 2039 ts_tablet_manager.cc:1939] T dfa5aab7c68644ff98276dff13ee2a90 P 84e1ed876b514410864d947cc44c25e6: Deleting consensus metadata
I20260929 14:47:25.823138 1495 catalog_manager.cc:4841] TS 827f714b0e7a4fcb96aa5fa06abb9b1d (127.0.98.2:34159): tablet dfa5aab7c68644ff98276dff13ee2a90 (table test-table [id=2e4f389a367f436190cbe4ef595ca931]) successfully deleted
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:25.823695 1493 catalog_manager.cc:4841] TS 84e1ed876b514410864d947cc44c25e6 (127.0.98.3:42693): tablet dfa5aab7c68644ff98276dff13ee2a90 (table test-table [id=2e4f389a367f436190cbe4ef595ca931]) successfully deleted
I20260929 14:47:25.828336 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.816496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.0.98.1:42799)
0929 14:47:25.816599 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.816604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.817080 (+ 476us) server_negotiation.cc:407] Connection header received
0929 14:47:25.817257 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.817260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.817303 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.817351 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.818256 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.819017 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.820236 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.820565 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.820633 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.824297 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.824309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.824325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.824351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.826414 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.826924 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.826929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.826931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.827031 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.827310 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.827313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.827315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.827603 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.827791 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.828008 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.828123 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 14:47:25.828728 1703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52074
I20260929 14:47:25.839195 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:25.830361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.62:33037)
0929 14:47:25.830488 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:25.830492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:25.830902 (+ 410us) server_negotiation.cc:407] Connection header received
0929 14:47:25.830994 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:25.830997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:25.831049 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:25.831133 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:25.831879 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.832710 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.833840 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:25.834214 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:25.834328 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:25.835805 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:25.835809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:25.835812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:25.835838 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:25.837474 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.837923 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.837926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.837928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.838012 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:25.838287 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:25.838290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:25.838292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:25.838495 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:25.838617 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:25.838920 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 14:47:25.839031 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 14:47:25.839531 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056
I20260929 14:47:25.840878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1580
I20260929 14:47:25.848620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1720
I20260929 14:47:25.855782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1860
I20260929 14:47:25.861219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 1476
2026-09-29T14:47:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2799 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[2077](info): set up 2 sockets
Sep 29 14:47:25 dist-test-slave-5hd1 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260929 14:47:26.900102 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:26 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:27.033082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:27.145256 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:27.145533 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:27.145574 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:27.149700 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:27.149763 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:47:27.149780 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:27.149799 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:27.149816 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:27.149857 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:27.155408 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:27.156561 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:27.157929 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:27.168350 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.168354 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.168481 2090 server_base.cc:1034] running on GCE node
W20260929 14:47:27.168794 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.169633 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:27.170888 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:27.172065 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790693247172022 us; error 63 us; skew 500 ppm
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:27.175521 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:27.176664 2090 webserver.cc:460] Webserver started at http://127.0.98.62:36377/ using document root <none> and password file <none>
I20260929 14:47:27.177281 2090 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:27.177349 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:27.177596 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:27.179807 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "7100f718a2eb47c39e5f51cf230ddeb9"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "8816189588d573dafffe4187d8118899"
server_key_iv: "a6c59962460a0ae21ce485b667579f49"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.180408 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "7100f718a2eb47c39e5f51cf230ddeb9"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "8816189588d573dafffe4187d8118899"
server_key_iv: "a6c59962460a0ae21ce485b667579f49"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.184521 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:47:27.187219 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.188763 2090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:27.188915 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "7100f718a2eb47c39e5f51cf230ddeb9"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "8816189588d573dafffe4187d8118899"
server_key_iv: "a6c59962460a0ae21ce485b667579f49"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.189034 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:27.209723 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:27.213104 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:27.213315 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:27.222498 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35675
I20260929 14:47:27.222616 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35675 every 8 connection(s)
I20260929 14:47:27.223548 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:27.227144 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:27.230091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2090
I20260929 14:47:27.230197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:27.230544 392 external_mini_cluster.cc:1428] Setting key a23c32bfa2ff59f0d5d46badf23ba2b3
I20260929 14:47:27.235189 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Bootstrap starting.
I20260929 14:47:27.238479 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:27.239588 2168 log.cc:824] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:27.242443 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: No bootstrap required, opened a new log
I20260929 14:47:27.247376 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260929 14:47:27.247717 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:27.247779 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7100f718a2eb47c39e5f51cf230ddeb9, State: Initialized, Role: FOLLOWER
I20260929 14:47:27.248389 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260929 14:47:27.248564 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:27.248636 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:27.248672 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.232425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.0.98.62:35675)
0929 14:47:27.232947 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.232970 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.233026 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:47:27.233982 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.234012 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.234536 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.234983 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.235833 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.237415 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.238365 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.239121 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.239311 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.242021 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.242031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.242053 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.242086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.245149 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.245608 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.245615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.245622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.245776 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.246049 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.246055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.246057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.246671 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.246894 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.247110 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.247472 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":128,"threads_started":1}
I20260929 14:47:27.248764 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:27.250102 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260929 14:47:27.250525 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7100f718a2eb47c39e5f51cf230ddeb9; no voters:
I20260929 14:47:27.250922 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:27.251070 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:27.251346 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [term 1 LEADER]: Becoming Leader. State: Replica: 7100f718a2eb47c39e5f51cf230ddeb9, State: Running, Role: LEADER
I20260929 14:47:27.251785 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } }
I20260929 14:47:27.252594 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:27.254074 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260929 14:47:27.254256 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:27.254077 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7100f718a2eb47c39e5f51cf230ddeb9. Latest consensus state: current_term: 1 leader_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7100f718a2eb47c39e5f51cf230ddeb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35675 } } }
I20260929 14:47:27.254702 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9 [sys.catalog]: This master's current role is: LEADER
W20260929 14:47:27.258792 2188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:27.258872 2188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:27.258951 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:27.259776 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:27.266708 2182 catalog_manager.cc:1347] Generated new cluster ID: ace54e8f70dd4551b89ed1a8c795f242
I20260929 14:47:27.266798 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:27.284865 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:27.286327 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:27.292372 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Generated new TSK 0
I20260929 14:47:27.293298 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:27.328256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35675
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:27.441627 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:27.441912 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:27.441992 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:27.446071 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:27.446159 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:27.446224 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:47:27.446272 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:27.446336 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:27.451515 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:27.452843 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:27.454067 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:27.463783 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.463939 2194 server_base.cc:1034] running on GCE node
W20260929 14:47:27.463802 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.464107 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.464624 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:27.465283 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:27.466439 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790693247466425 us; error 31 us; skew 500 ppm
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:27.469121 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:27.469929 2194 webserver.cc:460] Webserver started at http://127.0.98.1:34273/ using document root <none> and password file <none>
I20260929 14:47:27.470470 2194 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:27.470528 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:27.470760 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:27.472754 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "f57d4f715bed4a9691ae9b93d51feb48"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "27a245fc3e65f77202ac8fa436737543"
server_key_iv: "58a3bab508da194913fc44f53692a18c"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.473289 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f57d4f715bed4a9691ae9b93d51feb48"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "27a245fc3e65f77202ac8fa436737543"
server_key_iv: "58a3bab508da194913fc44f53692a18c"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.476912 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:27.479336 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.480607 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:27.480731 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "f57d4f715bed4a9691ae9b93d51feb48"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "27a245fc3e65f77202ac8fa436737543"
server_key_iv: "58a3bab508da194913fc44f53692a18c"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.480839 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:27.496920 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:27.500274 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:27.500469 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:27.501960 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:27.502035 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.502099 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:27.502148 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:27.513314 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40667
I20260929 14:47:27.513381 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40667 every 8 connection(s)
I20260929 14:47:27.514405 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:27.515128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2194
I20260929 14:47:27.515244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:27.515506 392 external_mini_cluster.cc:1428] Setting key 0d886fd6144fdd582886a58e1c595f69
I20260929 14:47:27.517851 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.502834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52981 (local address 127.0.98.62:35675)
0929 14:47:27.502969 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.502973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.504085 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 14:47:27.505898 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.505905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.505975 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.506075 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.507986 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.508942 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.510268 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.510673 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.510794 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.513648 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.513652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.513655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.513682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.515521 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.516082 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.516086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.516088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.516206 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.516677 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.516680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.516682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.516904 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.517008 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.517622 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.517734 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:27.519083 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.503443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.1:52981)
0929 14:47:27.503915 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.503956 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.505659 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.506531 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.506542 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.507080 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.507812 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.507832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.509089 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.509094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.510141 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.510149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.510825 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.510828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.510965 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.511613 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.511637 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.513453 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.515643 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.515648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.515663 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.515965 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.516292 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.516297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.516300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.516559 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.517037 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.517043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.517336 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.517585 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.517962 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260929 14:47:27.520536 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35675
I20260929 14:47:27.520687 2325 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:27.521258 2325 heartbeater.cc:511] Master 127.0.98.62:35675 requested a full tablet report, sending...
I20260929 14:47:27.523136 2119 ts_manager.cc:194] Registered new tserver with Master: f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667)
I20260929 14:47:27.524659 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52981
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:27.530838 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.522864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59807 (local address 127.0.98.62:35675)
0929 14:47:27.522978 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.522982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.523268 (+ 286us) server_negotiation.cc:407] Connection header received
0929 14:47:27.523381 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.523387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.523463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.523567 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.524089 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.524982 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.525883 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.526117 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.526224 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.527369 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.527373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.527376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.527398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.528975 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.529546 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.529549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.529550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.529623 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.529987 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.529990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.529991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.530174 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.530262 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.530608 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.530712 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260929 14:47:27.531014 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.522799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.1:59807)
0929 14:47:27.523159 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.523177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.523285 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.523605 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.523608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.523830 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.523955 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.523962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.525126 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.525129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.525756 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.525766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.526231 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.526234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.526324 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.526896 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.526908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.527231 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.529138 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.529141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.529143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.529421 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.529718 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.529724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.529729 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.529839 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.530278 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.530284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.530406 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.530528 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.530712 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:27.547169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35675
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:27.656702 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:27.656980 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:27.657069 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:27.661245 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:27.661307 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:27.661384 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:47:27.661418 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:27.661479 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:27.666740 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:27.667814 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:27.668897 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:27.678668 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.678659 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.678659 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.679080 2334 server_base.cc:1034] running on GCE node
I20260929 14:47:27.679594 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:27.680325 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:27.681492 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790693247681464 us; error 46 us; skew 500 ppm
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:27.684693 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:27.685655 2334 webserver.cc:460] Webserver started at http://127.0.98.2:34819/ using document root <none> and password file <none>
I20260929 14:47:27.686242 2334 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:27.686309 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:27.686597 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:27.688812 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "0735d20ad1015be2db225a56449f1673"
server_key_iv: "32c7161dac6a1c589cc07c854417c7f8"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.689420 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "0735d20ad1015be2db225a56449f1673"
server_key_iv: "32c7161dac6a1c589cc07c854417c7f8"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.693427 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 14:47:27.695931 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.697271 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:47:27.697453 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "0735d20ad1015be2db225a56449f1673"
server_key_iv: "32c7161dac6a1c589cc07c854417c7f8"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.697575 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:27.730935 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:27.734274 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:27.734499 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:27.735946 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:27.736021 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.736088 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:27.736131 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:27.747108 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45663
I20260929 14:47:27.747177 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45663 every 8 connection(s)
I20260929 14:47:27.748131 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:27.749675 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.736861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56599 (local address 127.0.98.62:35675)
0929 14:47:27.737013 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.737018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.737884 (+ 866us) server_negotiation.cc:407] Connection header received
0929 14:47:27.738983 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.738987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.739077 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.739144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.740607 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.741276 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.742419 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.742636 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.742749 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.745832 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.745835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.745837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.745859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.747464 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.747965 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.747968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.747970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.748056 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.748458 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.748465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.748468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.748709 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.748802 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.749438 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.749547 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:47:27.750355 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.737269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.2:56599)
0929 14:47:27.737713 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.737763 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.738725 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.739467 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.739474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.739811 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.740447 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.740471 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.741405 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.741409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.742295 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.742304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.742799 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.742803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.743002 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.743697 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.743713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.745654 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.747579 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.747585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.747599 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.747847 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.748150 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.748155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.748156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.748363 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.748851 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.748857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.749141 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.749299 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.749538 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":119,"threads_started":1}
I20260929 14:47:27.751564 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35675
I20260929 14:47:27.751703 2465 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:27.752257 2465 heartbeater.cc:511] Master 127.0.98.62:35675 requested a full tablet report, sending...
I20260929 14:47:27.753311 2119 ts_manager.cc:194] Registered new tserver with Master: 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663)
I20260929 14:47:27.753582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2334
I20260929 14:47:27.753679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:27.753952 392 external_mini_cluster.cc:1428] Setting key 2d1ff820fb2b71c8f108707c6eb53c59
I20260929 14:47:27.754068 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56599
I20260929 14:47:27.762113 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.754330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57655 (local address 127.0.98.62:35675)
0929 14:47:27.754438 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.754441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.754453 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:47:27.754484 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.754487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.754527 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.754615 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.755081 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.755795 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.757123 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.757351 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.757410 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.758786 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.758789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.758791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.758811 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.760366 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.760889 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.760892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.760894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.760970 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.761290 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.761292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.761293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.761506 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.761598 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.761902 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.761996 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:47:27.762339 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.753978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.2:57655)
0929 14:47:27.754264 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.754281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.754386 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.754644 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.754647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.754842 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.754940 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.754945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.755940 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.755946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.756961 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.756975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.757526 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.757528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.757595 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.758156 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.758175 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.758655 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.760470 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.760476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.760481 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.760756 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.761078 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.761085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.761089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.761194 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.761619 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.761622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.761726 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.761844 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.762002 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:27.781020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35675
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:27.889163 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:27.889405 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:27.889451 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:27.893425 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:27.893477 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:27.893515 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:47:27.893571 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:27.893661 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:27.898761 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:27.899760 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:27.900758 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:27.910008 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.910095 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:27.910003 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:27.910187 2474 server_base.cc:1034] running on GCE node
I20260929 14:47:27.910678 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:27.911332 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:27.912503 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790693247912490 us; error 28 us; skew 500 ppm
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:27.915087 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:27.915928 2474 webserver.cc:460] Webserver started at http://127.0.98.3:46083/ using document root <none> and password file <none>
I20260929 14:47:27.916488 2474 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:27.916555 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:27.916712 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:27.918592 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "54c2a0e72460478db9505fc48e02b0cf"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "08f814f59c77b0b49565370306f2a4c8"
server_key_iv: "b508c5017ca9bf0c8836a954e7005d55"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.919064 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54c2a0e72460478db9505fc48e02b0cf"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "08f814f59c77b0b49565370306f2a4c8"
server_key_iv: "b508c5017ca9bf0c8836a954e7005d55"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.922616 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:47:27.924897 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.926052 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:27.926169 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "54c2a0e72460478db9505fc48e02b0cf"
format_stamp: "Formatted at 2026-09-29 14:47:27 on dist-test-slave-5hd1"
server_key: "08f814f59c77b0b49565370306f2a4c8"
server_key_iv: "b508c5017ca9bf0c8836a954e7005d55"
server_key_version: "encryptionkey@0"
I20260929 14:47:27.926268 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:27.952952 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:27.956079 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:27.956291 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:27.957759 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:27.957804 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:27.957865 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:27.957897 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:27.969219 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38589
I20260929 14:47:27.969511 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38589 every 8 connection(s)
I20260929 14:47:27.970301 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:27.972097 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.958634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39977 (local address 127.0.98.62:35675)
0929 14:47:27.958759 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.958763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.959788 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 14:47:27.960860 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.960863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.960927 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.961027 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.962328 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.962847 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.964275 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.964580 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.964693 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.968331 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.968334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.968336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.968358 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.970061 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.970653 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.970657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.970659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.970751 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.971171 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.971177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.971178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.971381 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.971457 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.971883 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.971973 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:47:27.972886 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.959164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.3:39977)
0929 14:47:27.959637 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.959675 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.960651 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.961222 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.961233 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.961627 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.962185 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.962200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.962984 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.962987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.964107 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.964117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.964739 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.964743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.964937 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.965591 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.965617 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.968118 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.970186 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.970191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.970201 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.970534 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.970870 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.970873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.970875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.971052 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.971475 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.971480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.971682 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.971823 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.972057 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260929 14:47:27.974013 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35675
I20260929 14:47:27.974138 2605 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:27.974643 2605 heartbeater.cc:511] Master 127.0.98.62:35675 requested a full tablet report, sending...
I20260929 14:47:27.975632 2122 ts_manager.cc:194] Registered new tserver with Master: 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:27.976379 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39977
I20260929 14:47:27.977336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2474
I20260929 14:47:27.977464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:27.977797 392 external_mini_cluster.cc:1428] Setting key 22d23edfb65d9a9ebf4f1d292cd88ee2
I20260929 14:47:27.979707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:47:27.983497 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.976383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47205 (local address 127.0.98.62:35675)
0929 14:47:27.976496 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.976500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.976514 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:27.976636 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.976641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.976686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.976773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.977256 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.977994 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.979044 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.979237 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.979310 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.980493 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:27.980496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:27.980498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:27.980520 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:27.981910 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.982318 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.982323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.982325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.982400 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:27.982670 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:27.982674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:27.982675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:27.982854 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:27.982963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:27.983229 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 14:47:27.983349 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:27.983512 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.976165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.98.3:47205)
0929 14:47:27.976444 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:27.976460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:27.976544 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:27.976789 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:27.976791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:27.976972 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:27.977130 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.977138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.978128 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.978131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.978944 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:27.978956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.979340 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.979344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:27.979434 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:27.979944 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:27.979958 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:27.980351 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:27.982033 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.982037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.982039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.982223 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.982488 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:27.982493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:27.982494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:27.982577 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:27.982952 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:27.982955 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:27.983042 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 14:47:27.983179 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 14:47:27.983314 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":18,"thread_start_us":76,"threads_started":1}
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:47:27 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:28.004549 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:27.997013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.62:35675)
0929 14:47:27.997132 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:27.997135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:27.997148 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:27.997186 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:27.997188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:27.997224 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:27.997296 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:27.997935 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.998454 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.999262 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:27.999474 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:27.999565 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.001859 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:28.001862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:28.001864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:28.001897 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:28.003181 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:28.003544 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:28.003547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:28.003548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:28.003616 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:28.003838 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:28.003840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:28.003841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:28.004006 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:28.004105 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.004335 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.004432 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:47:28.007220 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:28.010315 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:28.024026 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.019037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.0.98.2:45663)
0929 14:47:28.019650 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.019665 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.019750 (+ 85us) server_negotiation.cc:407] Connection header received
0929 14:47:28.019846 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.019850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.020006 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.020170 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:28.020493 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.021394 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.022804 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.023379 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.023505 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.023686 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.023790 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.023878 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":104,"threads_started":1}
I20260929 14:47:28.024015 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.018552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45663 (local address 127.0.0.1:44684)
0929 14:47:28.019629 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.019644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.019752 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.020154 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.020160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.020206 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:28.020339 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.020348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.021539 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.021542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.022685 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.022693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.023501 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.023504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.023604 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.023638 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.023755 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.023873 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":91,"threads_started":1}
I20260929 14:47:28.024767 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.018990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38589 (local address 127.0.0.1:38094)
0929 14:47:28.019791 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.019805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.019879 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.020537 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.020540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.020567 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:28.020731 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.020739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.022223 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.022226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.023347 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.023354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.024376 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.024382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.024478 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.024503 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.024555 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.024628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":91,"threads_started":1}
I20260929 14:47:28.025089 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.018274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.0.1:52836)
0929 14:47:28.018663 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.018711 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.018830 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.020504 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.020507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.020524 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:28.020636 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.020642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.021970 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.021974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.023206 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.023216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.024695 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.024699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.024783 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.024807 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.024871 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.024942 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260929 14:47:28.025482 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.019286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.3:38589)
0929 14:47:28.019992 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.020004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.020026 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:28.020110 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.020115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.020393 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.020863 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:28.020992 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.022110 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.023473 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.024246 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.025038 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.025207 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.025255 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.025339 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":147,"threads_started":1}
I20260929 14:47:28.026963 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (DEFAULT_TABLE table=test-table [id=c2c482b89fc14ca7911a225852fc9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.027459 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (DEFAULT_TABLE table=test-table [id=c2c482b89fc14ca7911a225852fc9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.027881 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.028231 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.028676 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.018434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52836 (local address 127.0.98.1:40667)
0929 14:47:28.020086 (+ 1652us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.020098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.020117 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:47:28.020203 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.020208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.020372 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.020528 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:28.020728 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.021846 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.023358 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.024585 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.024684 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.028386 (+ 3702us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.028445 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.028546 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":94,"threads_started":1}
I20260929 14:47:28.031349 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (DEFAULT_TABLE table=test-table [id=c2c482b89fc14ca7911a225852fc9cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.032428 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.035544 2628 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Bootstrap starting.
I20260929 14:47:28.035686 2629 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Bootstrap starting.
I20260929 14:47:28.038084 2629 tablet_bootstrap.cc:654] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.038091 2628 tablet_bootstrap.cc:654] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.038692 2630 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Bootstrap starting.
I20260929 14:47:28.038995 2629 log.cc:824] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:28.038995 2628 log.cc:824] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:28.040928 2630 tablet_bootstrap.cc:654] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.041035 2628 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: No bootstrap required, opened a new log
I20260929 14:47:28.041035 2629 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: No bootstrap required, opened a new log
I20260929 14:47:28.041229 2628 ts_tablet_manager.cc:1397] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:47:28.041229 2629 ts_tablet_manager.cc:1397] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:47:28.041970 2630 log.cc:824] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:28.043470 2630 tablet_bootstrap.cc:492] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: No bootstrap required, opened a new log
I20260929 14:47:28.043666 2630 ts_tablet_manager.cc:1397] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:47:28.045222 2629 raft_consensus.cc:348] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.045222 2628 raft_consensus.cc:348] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.045487 2629 raft_consensus.cc:374] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.045487 2628 raft_consensus.cc:374] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.045544 2629 raft_consensus.cc:729] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.045544 2628 raft_consensus.cc:729] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d242017bfef419dbe0bafed9ebbf6b1, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.046005 2628 consensus_queue.cc:260] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.046005 2629 consensus_queue.cc:260] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.046663 2465 heartbeater.cc:503] Master 127.0.98.62:35675 was elected leader, sending a full tablet report...
I20260929 14:47:28.046671 2630 raft_consensus.cc:348] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.046932 2630 raft_consensus.cc:374] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.046989 2630 raft_consensus.cc:729] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.046998 2629 ts_tablet_manager.cc:1428] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:28.047300 2605 heartbeater.cc:503] Master 127.0.98.62:35675 was elected leader, sending a full tablet report...
I20260929 14:47:28.047400 2630 consensus_queue.cc:260] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.047997 2628 ts_tablet_manager.cc:1428] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:47:28.048852 2325 heartbeater.cc:503] Master 127.0.98.62:35675 was elected leader, sending a full tablet report...
I20260929 14:47:28.049072 2630 ts_tablet_manager.cc:1428] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:47:28.051533 2636 raft_consensus.cc:482] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:28.051671 2636 raft_consensus.cc:504] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.052809 2636 leader_election.cc:288] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589), 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663)
I20260929 14:47:28.057173 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.053015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33279 (local address 127.0.98.3:38589)
0929 14:47:28.053386 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.053392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.053406 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:28.053517 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.053520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.053572 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.053619 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:28.054047 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.054585 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056153 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.056608 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056731 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.056789 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.057003 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.057087 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260929 14:47:28.057183 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.053134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53629 (local address 127.0.98.2:45663)
0929 14:47:28.053261 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.053267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.053410 (+ 143us) server_negotiation.cc:407] Connection header received
0929 14:47:28.053527 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.053529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.053572 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.053644 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:28.053939 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.054491 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056261 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.056656 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056771 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.056830 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.057018 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.057090 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:47:28.057202 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.053052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45663 (local address 127.0.98.1:53629)
0929 14:47:28.053315 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.053330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.053435 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.053650 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.053652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.053664 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:28.053826 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.053832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.054601 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.054607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.056154 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.056163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.056771 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.056880 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.056906 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.056979 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.057051 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":93,"thread_start_us":105,"threads_started":1}
I20260929 14:47:28.057559 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.052938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38589 (local address 127.0.98.1:33279)
0929 14:47:28.053262 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.053277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.053367 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.053757 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.053760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.053778 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:28.053938 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.053944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.054666 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.054669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.056051 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.056061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.056732 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.056736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.056833 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.056864 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.057340 (+ 476us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.057428 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":121,"thread_start_us":101,"threads_started":1}
I20260929 14:47:28.057739 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e43833e8d2040b5b7f3b7d54d13b3c9" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" is_pre_election: true
I20260929 14:47:28.057997 2419 raft_consensus.cc:2454] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.058064 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e43833e8d2040b5b7f3b7d54d13b3c9" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf" is_pre_election: true
I20260929 14:47:28.058292 2559 raft_consensus.cc:2454] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.058461 2212 leader_election.cc:302] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d242017bfef419dbe0bafed9ebbf6b1, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.058750 2636 raft_consensus.cc:2781] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:28.058818 2636 raft_consensus.cc:482] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:28.058848 2636 raft_consensus.cc:3037] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.059814 2636 raft_consensus.cc:504] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.060197 2636 leader_election.cc:288] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Requested vote from peers 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589), 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663)
I20260929 14:47:28.060456 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e43833e8d2040b5b7f3b7d54d13b3c9" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf"
I20260929 14:47:28.060582 2559 raft_consensus.cc:3037] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.060590 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e43833e8d2040b5b7f3b7d54d13b3c9" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
I20260929 14:47:28.060720 2419 raft_consensus.cc:3037] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.061625 2559 raft_consensus.cc:2454] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.061973 2213 leader_election.cc:302] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c2a0e72460478db9505fc48e02b0cf, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.062095 2419 raft_consensus.cc:2454] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.062168 2636 raft_consensus.cc:2781] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:28.062398 2636 raft_consensus.cc:686] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Becoming Leader. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Running, Role: LEADER
I20260929 14:47:28.062763 2636 consensus_queue.cc:237] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } }
I20260929 14:47:28.065845 2119 catalog_manager.cc:5495] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:47:28 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:28.122169 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.113843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35675 (local address 127.0.0.1:53414)
0929 14:47:28.114197 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.114213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.114389 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.114636 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.114639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.115069 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:28.115152 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.115158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.115898 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.115905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.116548 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.116557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.116983 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.116986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.117062 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.117590 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:28.117602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:28.118953 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:28.120582 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:28.120631 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:28.120644 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:28.120857 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:28.121104 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:28.121107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:28.121108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:28.121180 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:28.121529 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:28.121535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:28.121645 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.121774 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.121956 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
I20260929 14:47:28.122169 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.114016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.0.98.62:35675)
0929 14:47:28.114180 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.114184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.114302 (+ 118us) server_negotiation.cc:407] Connection header received
0929 14:47:28.114503 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.114505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.114540 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.114621 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:28.115277 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.115780 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.116665 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.116873 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.116983 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.119081 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:28.119084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:28.119086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:28.119106 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:28.120486 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:28.120947 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:28.120949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:28.120950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:28.121022 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:28.121257 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:28.121259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:28.121260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:28.121430 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:28.121506 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.121839 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.122016 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":77}
I20260929 14:47:28.126082 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53414:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:47:28.126784 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:28.131736 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc81ff8c52475fa73e9c0900adcf7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22368cefa3cd45c594779e5d159c5db5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:28.131739 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc81ff8c52475fa73e9c0900adcf7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22368cefa3cd45c594779e5d159c5db5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:28.131974 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc81ff8c52475fa73e9c0900adcf7e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=22368cefa3cd45c594779e5d159c5db5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:28.132529 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc81ff8c52475fa73e9c0900adcf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.132529 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc81ff8c52475fa73e9c0900adcf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.132578 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc81ff8c52475fa73e9c0900adcf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.134843 2629 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: Bootstrap starting.
I20260929 14:47:28.135294 2630 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: Bootstrap starting.
I20260929 14:47:28.135684 2628 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: Bootstrap starting.
I20260929 14:47:28.135870 2629 tablet_bootstrap.cc:654] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.136698 2630 tablet_bootstrap.cc:654] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.136749 2628 tablet_bootstrap.cc:654] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.137308 2629 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: No bootstrap required, opened a new log
I20260929 14:47:28.137411 2629 ts_tablet_manager.cc:1397] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:28.137912 2630 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: No bootstrap required, opened a new log
I20260929 14:47:28.137912 2629 raft_consensus.cc:348] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.137995 2630 ts_tablet_manager.cc:1397] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:28.138005 2628 tablet_bootstrap.cc:492] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: No bootstrap required, opened a new log
I20260929 14:47:28.138028 2629 raft_consensus.cc:374] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.138069 2628 ts_tablet_manager.cc:1397] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 14:47:28.138068 2629 raft_consensus.cc:729] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.138190 2629 consensus_queue.cc:260] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.138458 2629 ts_tablet_manager.cc:1428] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:47:28.138454 2630 raft_consensus.cc:348] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.138454 2628 raft_consensus.cc:348] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.138553 2628 raft_consensus.cc:374] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.138553 2630 raft_consensus.cc:374] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.138587 2628 raft_consensus.cc:729] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d242017bfef419dbe0bafed9ebbf6b1, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.138587 2630 raft_consensus.cc:729] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.138707 2628 consensus_queue.cc:260] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.138707 2630 consensus_queue.cc:260] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.138942 2630 ts_tablet_manager.cc:1428] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:28.138964 2628 ts_tablet_manager.cc:1428] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:28.138458 2634 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.139173 2634 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.139135 2636 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.139241 2636 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.138967 2635 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.139436 2635 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 14:47:28.221290 2606 tablet.cc:2369] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:47:28.249416 2466 tablet.cc:2369] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:28.255283 2636 raft_consensus.cc:482] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:28.255466 2636 raft_consensus.cc:504] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.255931 2636 leader_election.cc:288] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:28.256377 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc81ff8c52475fa73e9c0900adcf7e" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" is_pre_election: true
I20260929 14:47:28.256565 2419 raft_consensus.cc:2454] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.256587 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc81ff8c52475fa73e9c0900adcf7e" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf" is_pre_election: true
I20260929 14:47:28.256774 2559 raft_consensus.cc:2454] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.256832 2212 leader_election.cc:302] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d242017bfef419dbe0bafed9ebbf6b1, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.257059 2636 raft_consensus.cc:2781] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:28.257134 2636 raft_consensus.cc:482] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:28.257164 2636 raft_consensus.cc:3037] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.258129 2636 raft_consensus.cc:504] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.258394 2636 leader_election.cc:288] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Requested vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:28.258716 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc81ff8c52475fa73e9c0900adcf7e" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
I20260929 14:47:28.258775 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc81ff8c52475fa73e9c0900adcf7e" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf"
I20260929 14:47:28.258838 2419 raft_consensus.cc:3037] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.258921 2559 raft_consensus.cc:3037] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.259827 2419 raft_consensus.cc:2454] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.260010 2559 raft_consensus.cc:2454] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.260123 2212 leader_election.cc:302] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d242017bfef419dbe0bafed9ebbf6b1, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.260301 2636 raft_consensus.cc:2781] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:28.260414 2636 raft_consensus.cc:686] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Becoming Leader. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Running, Role: LEADER
I20260929 14:47:28.260546 2636 consensus_queue.cc:237] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.261216 2639 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: TxnStatusTablet state changed. Reason: New leader f57d4f715bed4a9691ae9b93d51feb48. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.261322 2639 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:28.261621 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:47:28.261848 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:47:28.262346 2122 catalog_manager.cc:5495] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } health_report { overall_health: UNKNOWN } } }
W20260929 14:47:28.265811 2326 tablet.cc:2369] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:28.297605 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.293156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.0.98.1:40667)
0929 14:47:28.293282 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.293286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.293333 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:47:28.293528 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.293532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.293601 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.293714 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:28.293944 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.294717 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.295588 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.295932 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.296034 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.296505 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:28.297090 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:28.297203 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.297327 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.297408 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:47:28.297631 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.293099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.0.1:52840)
0929 14:47:28.293257 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.293274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.293381 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.293729 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.293731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.293737 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:28.293825 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.293829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.294836 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.294839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.295462 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.295470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.296085 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.296088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.296158 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.296364 (+ 206us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:28.297204 (+ 840us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:28.297224 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.297316 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.297440 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:47:28.303122 2419 raft_consensus.cc:1261] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.303174 2559 raft_consensus.cc:1261] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.303849 2639 consensus_queue.cc:1035] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:28.304059 2636 consensus_queue.cc:1035] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:28.307473 2635 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: TxnStatusTablet state changed. Reason: New leader f57d4f715bed4a9691ae9b93d51feb48. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.307587 2635 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.308382 2634 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: TxnStatusTablet state changed. Reason: New leader f57d4f715bed4a9691ae9b93d51feb48. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.308498 2634 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.312879 2636 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.312999 2636 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:28.313102 2654 mvcc.cc:205] Tried to move back new op lower bound from 7334679545042345984 to 7334679544876490752. Current Snapshot: MvccSnapshot[applied={T|T < 7334679545042345984}]
I20260929 14:47:28.314003 2634 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.314158 2634 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 54c2a0e72460478db9505fc48e02b0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.318236 2636 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.318342 2636 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P f57d4f715bed4a9691ae9b93d51feb48: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:28.319747 2635 tablet_replica.cc:439] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } } }
I20260929 14:47:28.320768 2635 tablet_replica.cc:442] T 4fcc81ff8c52475fa73e9c0900adcf7e P 3d242017bfef419dbe0bafed9ebbf6b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:28.321909 2656 mvcc.cc:205] Tried to move back new op lower bound from 7334679545042345984 to 7334679544876490752. Current Snapshot: MvccSnapshot[applied={T|T < 7334679545042345984}]
I20260929 14:47:28.332114 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.328183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.98.1:40667)
0929 14:47:28.328316 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.328321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.328335 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:28.328381 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.328384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.328433 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.328507 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:28.328867 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.329515 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.330776 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.331160 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.331287 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.331562 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:28.331657 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:28.331743 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.331848 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.331914 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 14:47:28.345222 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.341459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36597 (local address 127.0.98.1:40667)
0929 14:47:28.341572 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:28.341578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:28.341593 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:47:28.341634 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:28.341637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:28.341674 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:28.341742 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:28.342055 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.342777 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.343884 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.344265 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.344394 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.344727 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:28.344802 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:28.344876 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:28.344988 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 14:47:28.345057 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":48}
I20260929 14:47:28.345647 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:28.341094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.98.1:36597)
0929 14:47:28.341398 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:28.341414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:28.341534 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:28.341756 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:28.341759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:28.341765 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:28.341921 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.341931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.342881 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.342887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.343763 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:28.343772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:28.344465 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:28.344467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:28.344540 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:28.344577 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:28.344889 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:28.344898 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 14:47:28.345395 (+ 497us) client_negotiation.cc:240] Negotiation successful
0929 14:47:28.345491 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":21,"thread_start_us":117,"threads_started":1}
I20260929 14:47:28.353958 2559 raft_consensus.cc:1261] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.354127 2419 raft_consensus.cc:1261] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.354403 2639 consensus_queue.cc:1035] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:28.354632 2636 consensus_queue.cc:1035] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:28.380852 2655 mvcc.cc:205] Tried to move back new op lower bound from 7334679545253896192 to 7334679544067756032. Current Snapshot: MvccSnapshot[applied={T|T < 7334679545253896192}]
I20260929 14:47:28.404706 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:28.404867 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:28.407015 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Sending DeleteTablet for 3 replicas of tablet 5e43833e8d2040b5b7f3b7d54d13b3c9
I20260929 14:47:28.407776 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38094
I20260929 14:47:28.407775 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 14:47:28.408309 2670 tablet_replica.cc:330] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: stopping tablet replica
I20260929 14:47:28.408530 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52836
I20260929 14:47:28.408556 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:47:28.408654 2670 raft_consensus.cc:2229] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:28.408846 2670 raft_consensus.cc:2258] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:47:28.408931 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:28.408958 2672 tablet_replica.cc:330] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: stopping tablet replica
I20260929 14:47:28.409191 2671 tablet_replica.cc:330] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: stopping tablet replica
I20260929 14:47:28.409207 2672 raft_consensus.cc:2229] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:28.409464 2671 raft_consensus.cc:2229] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:28.409540 2672 raft_consensus.cc:2258] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:28.409668 2671 raft_consensus.cc:2258] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:28.410844 2672 ts_tablet_manager.cc:1905] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:28.410996 2671 ts_tablet_manager.cc:1905] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:28.414088 2671 ts_tablet_manager.cc:1918] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:28.414187 2671 log.cc:1196] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/5e43833e8d2040b5b7f3b7d54d13b3c9
I20260929 14:47:28.414196 2672 ts_tablet_manager.cc:1918] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:28.414278 2672 log.cc:1196] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/5e43833e8d2040b5b7f3b7d54d13b3c9
I20260929 14:47:28.414529 2671 ts_tablet_manager.cc:1939] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 54c2a0e72460478db9505fc48e02b0cf: Deleting consensus metadata
I20260929 14:47:28.414637 2672 ts_tablet_manager.cc:1939] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P f57d4f715bed4a9691ae9b93d51feb48: Deleting consensus metadata
I20260929 14:47:28.415139 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 7b875dbb848c4964bd6779373455914c (DEFAULT_TABLE table=test-table [id=63403f1fffee49df9133cacd01446045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.415380 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b875dbb848c4964bd6779373455914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.416502 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7b875dbb848c4964bd6779373455914c (DEFAULT_TABLE table=test-table [id=63403f1fffee49df9133cacd01446045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.416743 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b875dbb848c4964bd6779373455914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.417424 2630 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Bootstrap starting.
I20260929 14:47:28.418399 2630 tablet_bootstrap.cc:654] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.418717 2628 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Bootstrap starting.
I20260929 14:47:28.419577 2628 tablet_bootstrap.cc:654] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.419608 2630 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: No bootstrap required, opened a new log
I20260929 14:47:28.419687 2630 ts_tablet_manager.cc:1397] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:28.420153 2630 raft_consensus.cc:348] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.420317 2630 raft_consensus.cc:374] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.420363 2630 raft_consensus.cc:729] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.420506 2630 consensus_queue.cc:260] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.420804 2630 ts_tablet_manager.cc:1428] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:28.420823 2109 catalog_manager.cc:4841] TS 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589): tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (table test-table [id=c2c482b89fc14ca7911a225852fc9cda]) successfully deleted
I20260929 14:47:28.420949 2628 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: No bootstrap required, opened a new log
I20260929 14:47:28.421005 2109 catalog_manager.cc:4841] TS f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667): tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (table test-table [id=c2c482b89fc14ca7911a225852fc9cda]) successfully deleted
I20260929 14:47:28.421043 2628 ts_tablet_manager.cc:1397] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:47:28.421487 2628 raft_consensus.cc:348] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.421600 2628 raft_consensus.cc:374] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.421633 2628 raft_consensus.cc:729] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d242017bfef419dbe0bafed9ebbf6b1, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.421739 2628 consensus_queue.cc:260] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.421978 2628 ts_tablet_manager.cc:1428] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:47:28.422137 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 7b875dbb848c4964bd6779373455914c (DEFAULT_TABLE table=test-table [id=63403f1fffee49df9133cacd01446045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:28.422452 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b875dbb848c4964bd6779373455914c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:28.424723 2629 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Bootstrap starting.
I20260929 14:47:28.425595 2629 tablet_bootstrap.cc:654] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:28.425727 2670 ts_tablet_manager.cc:1905] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:28.426844 2629 tablet_bootstrap.cc:492] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: No bootstrap required, opened a new log
I20260929 14:47:28.426934 2629 ts_tablet_manager.cc:1397] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:28.427383 2629 raft_consensus.cc:348] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.427543 2629 raft_consensus.cc:374] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:28.427598 2629 raft_consensus.cc:729] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Initialized, Role: FOLLOWER
I20260929 14:47:28.427733 2629 consensus_queue.cc:260] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.428032 2629 ts_tablet_manager.cc:1428] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:28.428932 2670 ts_tablet_manager.cc:1918] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:28.429055 2670 log.cc:1196] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/5e43833e8d2040b5b7f3b7d54d13b3c9
I20260929 14:47:28.429333 2670 ts_tablet_manager.cc:1939] T 5e43833e8d2040b5b7f3b7d54d13b3c9 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting consensus metadata
I20260929 14:47:28.430529 2108 catalog_manager.cc:4841] TS 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663): tablet 5e43833e8d2040b5b7f3b7d54d13b3c9 (table test-table [id=c2c482b89fc14ca7911a225852fc9cda]) successfully deleted
I20260929 14:47:28.668673 2669 raft_consensus.cc:482] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:28.668817 2669 raft_consensus.cc:504] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.669210 2669 leader_election.cc:288] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:28.669710 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b875dbb848c4964bd6779373455914c" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf" is_pre_election: true
I20260929 14:47:28.669709 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b875dbb848c4964bd6779373455914c" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" is_pre_election: true
I20260929 14:47:28.669840 2558 raft_consensus.cc:2454] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.669896 2419 raft_consensus.cc:2454] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:28.670192 2213 leader_election.cc:302] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c2a0e72460478db9505fc48e02b0cf, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.670396 2669 raft_consensus.cc:2781] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:28.670481 2669 raft_consensus.cc:482] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:28.670542 2669 raft_consensus.cc:3037] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.671645 2669 raft_consensus.cc:504] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.672046 2669 leader_election.cc:288] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Requested vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:28.672456 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b875dbb848c4964bd6779373455914c" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
I20260929 14:47:28.672453 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b875dbb848c4964bd6779373455914c" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf"
I20260929 14:47:28.672564 2419 raft_consensus.cc:3037] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.672597 2558 raft_consensus.cc:3037] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:28.673571 2419 raft_consensus.cc:2454] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.673611 2558 raft_consensus.cc:2454] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:28.673859 2212 leader_election.cc:302] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d242017bfef419dbe0bafed9ebbf6b1, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:28.674067 2669 raft_consensus.cc:2781] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:28.674185 2669 raft_consensus.cc:686] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Becoming Leader. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Running, Role: LEADER
I20260929 14:47:28.674331 2669 consensus_queue.cc:237] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:28.675963 2122 catalog_manager.cc:5495] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:28.736222 2558 raft_consensus.cc:1261] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.736222 2419 raft_consensus.cc:1261] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:28.736755 2636 consensus_queue.cc:1035] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:28.736956 2669 consensus_queue.cc:1035] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:47:28.754626 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:47:30.318832 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:30.319015 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:30.322292 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Sending DeleteTablet for 3 replicas of tablet 7b875dbb848c4964bd6779373455914c
I20260929 14:47:30.322997 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b875dbb848c4964bd6779373455914c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52836
I20260929 14:47:30.323268 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b875dbb848c4964bd6779373455914c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38094
I20260929 14:47:30.323437 2699 tablet_replica.cc:330] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: stopping tablet replica
I20260929 14:47:30.323592 2699 raft_consensus.cc:2229] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:30.323686 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b875dbb848c4964bd6779373455914c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 14:47:30.323843 2699 raft_consensus.cc:2258] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.324934 2699 ts_tablet_manager.cc:1905] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.328447 2701 tablet_replica.cc:330] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: stopping tablet replica
I20260929 14:47:30.328646 2701 raft_consensus.cc:2229] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:30.328802 2701 raft_consensus.cc:2258] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.329036 2699 ts_tablet_manager.cc:1918] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:47:30.329133 2699 log.cc:1196] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/7b875dbb848c4964bd6779373455914c
I20260929 14:47:30.329571 2699 ts_tablet_manager.cc:1939] T 7b875dbb848c4964bd6779373455914c P f57d4f715bed4a9691ae9b93d51feb48: Deleting consensus metadata
I20260929 14:47:30.329572 2701 ts_tablet_manager.cc:1905] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.329941 2700 tablet_replica.cc:330] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: stopping tablet replica
I20260929 14:47:30.330101 2700 raft_consensus.cc:2229] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:30.330248 2700 raft_consensus.cc:2258] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.330530 2109 catalog_manager.cc:4841] TS f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667): tablet 7b875dbb848c4964bd6779373455914c (table test-table [id=63403f1fffee49df9133cacd01446045]) successfully deleted
I20260929 14:47:30.331122 2700 ts_tablet_manager.cc:1905] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.333426 2701 ts_tablet_manager.cc:1918] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:47:30.333527 2701 log.cc:1196] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/7b875dbb848c4964bd6779373455914c
I20260929 14:47:30.333678 2700 ts_tablet_manager.cc:1918] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:47:30.333774 2700 log.cc:1196] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/7b875dbb848c4964bd6779373455914c
I20260929 14:47:30.333858 2701 ts_tablet_manager.cc:1939] T 7b875dbb848c4964bd6779373455914c P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting consensus metadata
I20260929 14:47:30.334147 2700 ts_tablet_manager.cc:1939] T 7b875dbb848c4964bd6779373455914c P 54c2a0e72460478db9505fc48e02b0cf: Deleting consensus metadata
I20260929 14:47:30.334713 2108 catalog_manager.cc:4841] TS 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663): tablet 7b875dbb848c4964bd6779373455914c (table test-table [id=63403f1fffee49df9133cacd01446045]) successfully deleted
I20260929 14:47:30.335057 2109 catalog_manager.cc:4841] TS 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589): tablet 7b875dbb848c4964bd6779373455914c (table test-table [id=63403f1fffee49df9133cacd01446045]) successfully deleted
Sep 29 14:47:30 dist-test-slave-5hd1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:47:30 dist-test-slave-5hd1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693250, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:30.363929 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.346513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.98.62:35675)
0929 14:47:30.346915 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.346923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.346942 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:47:30.346999 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.347002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.347058 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.347154 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:30.347844 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.348558 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.349449 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.349675 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.349755 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.360363 (+ 10608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:30.360367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:30.360370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:30.360407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:30.362138 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:30.362585 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:30.362588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:30.362589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:30.362675 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:30.362916 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:30.362918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:30.362920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:30.363152 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:30.363270 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.363509 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.363673 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260929 14:47:30.367153 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:30.367836 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:30.373301 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89e990dac048d08232cb86e1a29633 (DEFAULT_TABLE table=test-table [id=79b03372f75c4336985fa2a03d05ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.373493 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89e990dac048d08232cb86e1a29633 (DEFAULT_TABLE table=test-table [id=79b03372f75c4336985fa2a03d05ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.373651 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89e990dac048d08232cb86e1a29633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.373910 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89e990dac048d08232cb86e1a29633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.375932 2710 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Bootstrap starting.
I20260929 14:47:30.376241 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89e990dac048d08232cb86e1a29633 (DEFAULT_TABLE table=test-table [id=79b03372f75c4336985fa2a03d05ebb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.376559 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89e990dac048d08232cb86e1a29633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.377081 2710 tablet_bootstrap.cc:654] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.378772 2710 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: No bootstrap required, opened a new log
I20260929 14:47:30.378847 2710 ts_tablet_manager.cc:1397] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:30.379305 2710 raft_consensus.cc:348] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.379439 2710 raft_consensus.cc:374] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.379498 2710 raft_consensus.cc:729] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d242017bfef419dbe0bafed9ebbf6b1, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.379657 2710 consensus_queue.cc:260] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.379781 2711 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Bootstrap starting.
I20260929 14:47:30.380085 2710 ts_tablet_manager.cc:1428] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:30.380421 2713 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Bootstrap starting.
I20260929 14:47:30.380980 2711 tablet_bootstrap.cc:654] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.381419 2713 tablet_bootstrap.cc:654] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.382494 2711 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: No bootstrap required, opened a new log
I20260929 14:47:30.382596 2711 ts_tablet_manager.cc:1397] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:30.382596 2713 tablet_bootstrap.cc:492] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: No bootstrap required, opened a new log
I20260929 14:47:30.382700 2713 ts_tablet_manager.cc:1397] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:30.383157 2713 raft_consensus.cc:348] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.383184 2711 raft_consensus.cc:348] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.383283 2713 raft_consensus.cc:374] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.383320 2711 raft_consensus.cc:374] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.383322 2713 raft_consensus.cc:729] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.383364 2711 raft_consensus.cc:729] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.383502 2713 consensus_queue.cc:260] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.383538 2711 consensus_queue.cc:260] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.383806 2713 ts_tablet_manager.cc:1428] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:30.384029 2711 ts_tablet_manager.cc:1428] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:30.384820 2717 raft_consensus.cc:482] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:30.384961 2717 raft_consensus.cc:504] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.387017 2717 leader_election.cc:288] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667), 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663)
I20260929 14:47:30.391021 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.386540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41239 (local address 127.0.98.1:40667)
0929 14:47:30.387240 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.387251 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.387511 (+ 260us) server_negotiation.cc:407] Connection header received
0929 14:47:30.387663 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.387666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.387719 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.387818 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:30.388099 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.388766 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.389924 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.390395 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.390523 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.390605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.390786 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.390874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":86,"threads_started":1}
I20260929 14:47:30.391036 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.386479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.98.3:41239)
0929 14:47:30.387417 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:30.387434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:30.387555 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:30.387829 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:30.387832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:30.387849 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:30.387991 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.387997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.388882 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.388889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.389788 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.389797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.390518 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.390522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.390639 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.390663 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:47:30.390758 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 14:47:30.390861 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":121,"threads_started":1}
I20260929 14:47:30.391345 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.387315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36593 (local address 127.0.98.2:45663)
0929 14:47:30.387900 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.387906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.387916 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:47:30.387952 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.387954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.388002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.388099 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:30.388432 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.388997 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.390112 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.390547 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.390670 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.390733 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.390932 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.391010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":99,"threads_started":1}
I20260929 14:47:30.391592 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89e990dac048d08232cb86e1a29633" candidate_uuid: "54c2a0e72460478db9505fc48e02b0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d4f715bed4a9691ae9b93d51feb48" is_pre_election: true
I20260929 14:47:30.391819 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.387252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45663 (local address 127.0.98.3:36593)
0929 14:47:30.387642 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:30.387656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:30.387741 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:30.388111 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:30.388114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:30.388132 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:30.388322 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.388328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.389130 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.389133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.389980 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.389988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.390682 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.390686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.390791 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.390826 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:47:30.390918 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:47:30.391009 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":160,"threads_started":1}
I20260929 14:47:30.391920 2279 raft_consensus.cc:2454] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54c2a0e72460478db9505fc48e02b0cf in term 0.
I20260929 14:47:30.392316 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89e990dac048d08232cb86e1a29633" candidate_uuid: "54c2a0e72460478db9505fc48e02b0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" is_pre_election: true
I20260929 14:47:30.392467 2419 raft_consensus.cc:2454] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54c2a0e72460478db9505fc48e02b0cf in term 0.
I20260929 14:47:30.392448 2493 leader_election.cc:302] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c2a0e72460478db9505fc48e02b0cf, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:30.392791 2717 raft_consensus.cc:2781] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:30.392874 2717 raft_consensus.cc:482] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:30.392917 2717 raft_consensus.cc:3037] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.394045 2717 raft_consensus.cc:504] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.394405 2717 leader_election.cc:288] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [CANDIDATE]: Term 1 election: Requested vote from peers f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667), 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663)
I20260929 14:47:30.394763 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89e990dac048d08232cb86e1a29633" candidate_uuid: "54c2a0e72460478db9505fc48e02b0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
I20260929 14:47:30.394773 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89e990dac048d08232cb86e1a29633" candidate_uuid: "54c2a0e72460478db9505fc48e02b0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f57d4f715bed4a9691ae9b93d51feb48"
I20260929 14:47:30.394889 2419 raft_consensus.cc:3037] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.394901 2279 raft_consensus.cc:3037] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.395983 2419 raft_consensus.cc:2454] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54c2a0e72460478db9505fc48e02b0cf in term 1.
I20260929 14:47:30.395983 2279 raft_consensus.cc:2454] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54c2a0e72460478db9505fc48e02b0cf in term 1.
I20260929 14:47:30.396456 2493 leader_election.cc:302] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c2a0e72460478db9505fc48e02b0cf, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:30.396623 2717 raft_consensus.cc:2781] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:30.396850 2717 raft_consensus.cc:686] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 1 LEADER]: Becoming Leader. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Running, Role: LEADER
I20260929 14:47:30.397069 2717 consensus_queue.cc:237] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.399382 2122 catalog_manager.cc:5495] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54c2a0e72460478db9505fc48e02b0cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } health_report { overall_health: HEALTHY } } }
I20260929 14:47:30.416936 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.413409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.98.3:38589)
0929 14:47:30.413651 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.413658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.413669 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:47:30.413736 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.413738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.413785 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.413888 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:30.414106 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.414634 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.415430 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.415670 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.415743 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.415967 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.416509 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.416599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.416671 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.416714 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":122,"threads_started":1}
I20260929 14:47:30.426826 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.423427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53815 (local address 127.0.98.1:40667)
0929 14:47:30.423598 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.423603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.423760 (+ 157us) server_negotiation.cc:407] Connection header received
0929 14:47:30.423933 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.423936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.423998 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.424084 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:30.424365 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.424908 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.425765 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.425988 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.426099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.426393 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.426460 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.426540 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.426630 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.426680 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:47:30.426838 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.423293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.98.3:53815)
0929 14:47:30.423668 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:30.423681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:30.423808 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:30.424092 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:30.424095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:30.424102 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:30.424247 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.424253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.425026 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.425032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.425644 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.425652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.426100 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.426104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.426201 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.426248 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.426546 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.426554 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:47:30.426620 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:47:30.426695 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":104,"threads_started":1}
I20260929 14:47:30.432550 2419 raft_consensus.cc:1261] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Refusing update from remote peer 54c2a0e72460478db9505fc48e02b0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:30.432734 2279 raft_consensus.cc:1261] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Refusing update from remote peer 54c2a0e72460478db9505fc48e02b0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:30.433018 2722 consensus_queue.cc:1035] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:30.433175 2717 consensus_queue.cc:1035] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:30.450266 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.446484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38589 (local address 127.0.98.1:49515)
0929 14:47:30.446731 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:30.446749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:30.446851 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:30.447167 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:30.447169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:30.447175 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:30.447277 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.447281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.448083 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.448086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.448883 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:30.448892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.449577 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.449582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:30.449692 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.449726 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.450027 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.450033 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:47:30.450086 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 14:47:30.450154 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260929 14:47:30.450263 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.446543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49515 (local address 127.0.98.3:38589)
0929 14:47:30.446649 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.446654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.446828 (+ 174us) server_negotiation.cc:407] Connection header received
0929 14:47:30.447044 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.447047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.447091 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.447156 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:30.447430 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.447962 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.449008 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.449359 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.449458 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.449849 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.449927 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.450005 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.450117 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.450166 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 14:47:30.472803 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:30.472987 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:30.475126 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Sending DeleteTablet for 3 replicas of tablet 7f89e990dac048d08232cb86e1a29633
I20260929 14:47:30.475708 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89e990dac048d08232cb86e1a29633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52836
I20260929 14:47:30.475898 2699 tablet_replica.cc:330] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: stopping tablet replica
I20260929 14:47:30.475793 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89e990dac048d08232cb86e1a29633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38094
I20260929 14:47:30.475937 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89e990dac048d08232cb86e1a29633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 14:47:30.476008 2699 raft_consensus.cc:2229] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:30.476050 2700 tablet_replica.cc:330] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: stopping tablet replica
I20260929 14:47:30.476127 2699 raft_consensus.cc:2258] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.476191 2700 raft_consensus.cc:2229] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:30.476339 2701 tablet_replica.cc:330] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: stopping tablet replica
I20260929 14:47:30.476449 2701 raft_consensus.cc:2229] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:30.476469 2700 raft_consensus.cc:2258] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.476541 2701 raft_consensus.cc:2258] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:30.477504 2699 ts_tablet_manager.cc:1905] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.477432 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:30.477818 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:30.481884 2701 ts_tablet_manager.cc:1905] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.482573 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 (DEFAULT_TABLE table=test-table [id=a6635d488385459493929a2ee579ad1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.482736 2525 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 (DEFAULT_TABLE table=test-table [id=a6635d488385459493929a2ee579ad1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.482808 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c257f4c54d91b5061ba073960c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.482982 2525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c257f4c54d91b5061ba073960c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.484139 2701 ts_tablet_manager.cc:1918] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:30.484233 2701 log.cc:1196] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/7f89e990dac048d08232cb86e1a29633
I20260929 14:47:30.484512 2701 ts_tablet_manager.cc:1939] T 7f89e990dac048d08232cb86e1a29633 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting consensus metadata
I20260929 14:47:30.485299 2710 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Bootstrap starting.
I20260929 14:47:30.486011 2710 tablet_bootstrap.cc:654] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.486442 2711 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Bootstrap starting.
I20260929 14:47:30.486610 2108 catalog_manager.cc:4841] TS 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663): tablet 7f89e990dac048d08232cb86e1a29633 (table test-table [id=79b03372f75c4336985fa2a03d05ebb4]) successfully deleted
I20260929 14:47:30.486796 2700 ts_tablet_manager.cc:1905] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:30.487036 2699 ts_tablet_manager.cc:1918] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:30.487108 2699 log.cc:1196] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/7f89e990dac048d08232cb86e1a29633
I20260929 14:47:30.487352 2699 ts_tablet_manager.cc:1939] T 7f89e990dac048d08232cb86e1a29633 P f57d4f715bed4a9691ae9b93d51feb48: Deleting consensus metadata
I20260929 14:47:30.487439 2711 tablet_bootstrap.cc:654] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.488557 2711 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: No bootstrap required, opened a new log
I20260929 14:47:30.488611 2711 ts_tablet_manager.cc:1397] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:30.488898 2700 ts_tablet_manager.cc:1918] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:30.489063 2700 log.cc:1196] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/7f89e990dac048d08232cb86e1a29633
I20260929 14:47:30.488914 2711 raft_consensus.cc:348] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.489114 2711 raft_consensus.cc:374] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.489156 2711 raft_consensus.cc:729] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2a0e72460478db9505fc48e02b0cf, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.489322 2700 ts_tablet_manager.cc:1939] T 7f89e990dac048d08232cb86e1a29633 P 54c2a0e72460478db9505fc48e02b0cf: Deleting consensus metadata
I20260929 14:47:30.489323 2711 consensus_queue.cc:260] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.489661 2711 ts_tablet_manager.cc:1428] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:30.489666 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 (DEFAULT_TABLE table=test-table [id=a6635d488385459493929a2ee579ad1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:30.489890 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1c257f4c54d91b5061ba073960c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:30.492107 2713 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: Bootstrap starting.
I20260929 14:47:30.492272 2109 catalog_manager.cc:4841] TS 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589): tablet 7f89e990dac048d08232cb86e1a29633 (table test-table [id=79b03372f75c4336985fa2a03d05ebb4]) successfully deleted
I20260929 14:47:30.492396 2109 catalog_manager.cc:4841] TS f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667): tablet 7f89e990dac048d08232cb86e1a29633 (table test-table [id=79b03372f75c4336985fa2a03d05ebb4]) successfully deleted
I20260929 14:47:30.492982 2713 tablet_bootstrap.cc:654] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:30.493880 2710 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: No bootstrap required, opened a new log
I20260929 14:47:30.493973 2710 ts_tablet_manager.cc:1397] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 14:47:30.494165 2713 tablet_bootstrap.cc:492] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: No bootstrap required, opened a new log
I20260929 14:47:30.494263 2713 ts_tablet_manager.cc:1397] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:30.494418 2710 raft_consensus.cc:348] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.494585 2710 raft_consensus.cc:374] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.494639 2710 raft_consensus.cc:729] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d242017bfef419dbe0bafed9ebbf6b1, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.494616 2713 raft_consensus.cc:348] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.494724 2713 raft_consensus.cc:374] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:30.494760 2713 raft_consensus.cc:729] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Initialized, Role: FOLLOWER
I20260929 14:47:30.494768 2710 consensus_queue.cc:260] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.494886 2713 consensus_queue.cc:260] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.495000 2682 raft_consensus.cc:482] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:30.495034 2710 ts_tablet_manager.cc:1428] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:30.495085 2682 raft_consensus.cc:504] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.495147 2713 ts_tablet_manager.cc:1428] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:30.495420 2682 leader_election.cc:288] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:30.495868 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c257f4c54d91b5061ba073960c07" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf" is_pre_election: true
I20260929 14:47:30.495959 2558 raft_consensus.cc:2454] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:30.496253 2213 leader_election.cc:302] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54c2a0e72460478db9505fc48e02b0cf, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:30.496764 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c257f4c54d91b5061ba073960c07" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" is_pre_election: true
I20260929 14:47:30.496884 2419 raft_consensus.cc:2454] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 0.
I20260929 14:47:30.497335 2682 raft_consensus.cc:2781] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:30.497399 2682 raft_consensus.cc:482] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:30.497437 2682 raft_consensus.cc:3037] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.498435 2682 raft_consensus.cc:504] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.498773 2682 leader_election.cc:288] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Requested vote from peers 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663), 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589)
I20260929 14:47:30.499084 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c257f4c54d91b5061ba073960c07" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d242017bfef419dbe0bafed9ebbf6b1"
I20260929 14:47:30.499110 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1c257f4c54d91b5061ba073960c07" candidate_uuid: "f57d4f715bed4a9691ae9b93d51feb48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54c2a0e72460478db9505fc48e02b0cf"
I20260929 14:47:30.499202 2419 raft_consensus.cc:3037] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.499208 2558 raft_consensus.cc:3037] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:30.500001 2419 raft_consensus.cc:2454] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:30.500257 2558 raft_consensus.cc:2454] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f57d4f715bed4a9691ae9b93d51feb48 in term 1.
I20260929 14:47:30.500272 2212 leader_election.cc:302] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d242017bfef419dbe0bafed9ebbf6b1, f57d4f715bed4a9691ae9b93d51feb48; no voters:
I20260929 14:47:30.500455 2682 raft_consensus.cc:2781] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:30.500564 2682 raft_consensus.cc:686] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [term 1 LEADER]: Becoming Leader. State: Replica: f57d4f715bed4a9691ae9b93d51feb48, State: Running, Role: LEADER
I20260929 14:47:30.500715 2682 consensus_queue.cc:237] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } }
I20260929 14:47:30.502285 2119 catalog_manager.cc:5495] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f57d4f715bed4a9691ae9b93d51feb48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f57d4f715bed4a9691ae9b93d51feb48" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:30.516381 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:30.513297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.0.98.1:40667)
0929 14:47:30.513418 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:30.513422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:30.513437 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:47:30.513633 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:30.513641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:30.513682 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:30.513759 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:30.513999 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.514519 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.515329 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:30.515570 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:30.515679 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:30.515934 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:30.516012 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:30.516090 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:30.516200 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 14:47:30.516248 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:47:30.526858 2558 raft_consensus.cc:1261] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:30.527144 2419 raft_consensus.cc:1261] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Refusing update from remote peer f57d4f715bed4a9691ae9b93d51feb48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:30.527601 2636 consensus_queue.cc:1035] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d242017bfef419dbe0bafed9ebbf6b1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:30.527889 2636 consensus_queue.cc:1035] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54c2a0e72460478db9505fc48e02b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:47:30.542886 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:47:32.099346 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:32.099517 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:32.102064 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7100f718a2eb47c39e5f51cf230ddeb9: Sending DeleteTablet for 3 replicas of tablet 6cb1c257f4c54d91b5061ba073960c07
I20260929 14:47:32.102589 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52836
I20260929 14:47:32.102669 2525 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38094
I20260929 14:47:32.102741 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1c257f4c54d91b5061ba073960c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 14:47:32.102962 2762 tablet_replica.cc:330] T 6cb1c257f4c54d91b5061ba073960c07 P f57d4f715bed4a9691ae9b93d51feb48: stopping tablet replica
I20260929 14:47:32.102957 2763 tablet_replica.cc:330] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: stopping tablet replica
I20260929 14:47:32.103018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2194
I20260929 14:47:32.103092 2763 raft_consensus.cc:2229] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:32.104633 2763 raft_consensus.cc:2258] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:32.104873 2764 tablet_replica.cc:330] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: stopping tablet replica
I20260929 14:47:32.105010 2764 raft_consensus.cc:2229] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:32.105139 2764 raft_consensus.cc:2258] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:32.105726 2764 ts_tablet_manager.cc:1905] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:32.105726 2763 ts_tablet_manager.cc:1905] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:32.108265 2763 ts_tablet_manager.cc:1918] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:47:32.108268 2764 ts_tablet_manager.cc:1918] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:47:32.108343 2764 log.cc:1196] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/6cb1c257f4c54d91b5061ba073960c07
I20260929 14:47:32.108352 2763 log.cc:1196] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/6cb1c257f4c54d91b5061ba073960c07
I20260929 14:47:32.108619 2764 ts_tablet_manager.cc:1939] T 6cb1c257f4c54d91b5061ba073960c07 P 3d242017bfef419dbe0bafed9ebbf6b1: Deleting consensus metadata
I20260929 14:47:32.108625 2763 ts_tablet_manager.cc:1939] T 6cb1c257f4c54d91b5061ba073960c07 P 54c2a0e72460478db9505fc48e02b0cf: Deleting consensus metadata
I20260929 14:47:32.109404 2109 catalog_manager.cc:4841] TS 54c2a0e72460478db9505fc48e02b0cf (127.0.98.3:38589): tablet 6cb1c257f4c54d91b5061ba073960c07 (table test-table [id=a6635d488385459493929a2ee579ad1e]) successfully deleted
I20260929 14:47:32.109504 2108 catalog_manager.cc:4841] TS 3d242017bfef419dbe0bafed9ebbf6b1 (127.0.98.2:45663): tablet 6cb1c257f4c54d91b5061ba073960c07 (table test-table [id=a6635d488385459493929a2ee579ad1e]) successfully deleted
W20260929 14:47:32.111261 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40667 (error 108)
I20260929 14:47:32.111480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2334
I20260929 14:47:32.113068 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:32.112579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40667 (local address 127.0.0.1:38044)
0929 14:47:32.112844 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:32.112937 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
W20260929 14:47:32.113234 2109 catalog_manager.cc:4584] TS f57d4f715bed4a9691ae9b93d51feb48 (127.0.98.1:40667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cb1c257f4c54d91b5061ba073960c07: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40667: connect: Connection refused (error 111)
I20260929 14:47:32.118468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2474
I20260929 14:47:32.125106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2090
2026-09-29T14:47:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6264 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:32 dist-test-slave-5hd1 krb5kdc[2769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:32 dist-test-slave-5hd1 krb5kdc[2769](info): set up 2 sockets
Sep 29 14:47:32 dist-test-slave-5hd1 krb5kdc[2769](info): commencing operation
krb5kdc: starting...
W20260929 14:47:33.184733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:33.321893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38333
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:33.441256 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:33.441638 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:33.441726 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:33.446053 2782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:33.446188 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:33.446234 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:33.446309 2782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:33.446372 2782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:33.452603 2782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:33.453851 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:33.455130 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:33.464699 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:33.464699 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:33.464737 2782 server_base.cc:1034] running on GCE node
W20260929 14:47:33.464699 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:33.465575 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:33.466677 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:33.467869 2782 hybrid_clock.cc:648] HybridClock initialized: now 1790693253467835 us; error 46 us; skew 500 ppm
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:33.471145 2782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:33.472460 2782 webserver.cc:460] Webserver started at http://127.0.98.62:40991/ using document root <none> and password file <none>
I20260929 14:47:33.473101 2782 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:33.473198 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:33.473448 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:33.475598 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "30ff2581d7b94dc4883c159a3b478e92"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "13ade0ae733d3730c7a6ce5de492c339"
server_key_iv: "cc04568b67eb51b67fccf246c489c774"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.476241 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "30ff2581d7b94dc4883c159a3b478e92"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "13ade0ae733d3730c7a6ce5de492c339"
server_key_iv: "cc04568b67eb51b67fccf246c489c774"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.480152 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 14:47:33.482695 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:33.483958 2782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:33.484105 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "30ff2581d7b94dc4883c159a3b478e92"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "13ade0ae733d3730c7a6ce5de492c339"
server_key_iv: "cc04568b67eb51b67fccf246c489c774"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.484261 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:33.500391 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:33.503487 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:33.503736 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:33.511637 2782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38333
I20260929 14:47:33.511649 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38333 every 8 connection(s)
I20260929 14:47:33.512661 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:33.515561 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:33.519111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2782
I20260929 14:47:33.519243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:33.519518 392 external_mini_cluster.cc:1428] Setting key 3987ca8459171d1aed8ce477ceb8e913
I20260929 14:47:33.521363 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: Bootstrap starting.
I20260929 14:47:33.523849 2850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:33.524668 2850 log.cc:824] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:33.526904 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: No bootstrap required, opened a new log
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:33.531579 2850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260929 14:47:33.531842 2850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:33.531903 2850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ff2581d7b94dc4883c159a3b478e92, State: Initialized, Role: FOLLOWER
I20260929 14:47:33.532455 2850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260929 14:47:33.532634 2850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:33.532692 2850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:33.532787 2850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:33.533968 2850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260929 14:47:33.534329 2850 leader_election.cc:302] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ff2581d7b94dc4883c159a3b478e92; no voters:
I20260929 14:47:33.534682 2850 leader_election.cc:288] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:33.534816 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:33.534838 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:33.521003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.98.62:38333)
0929 14:47:33.521392 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:33.521418 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:33.521453 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:47:33.522110 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:33.522127 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:33.522445 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:33.522717 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:33.523645 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.524845 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.525908 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.526388 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.526525 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:33.529330 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:33.529336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:33.529353 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:33.529382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:33.531425 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:33.532108 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:33.532114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:33.532120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:33.532276 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:33.532557 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:33.532561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:33.532562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:33.533101 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:33.533305 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:33.533539 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 14:47:33.533847 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":126,"threads_started":1}
I20260929 14:47:33.535072 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [term 1 LEADER]: Becoming Leader. State: Replica: 30ff2581d7b94dc4883c159a3b478e92, State: Running, Role: LEADER
I20260929 14:47:33.535389 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } }
I20260929 14:47:33.535924 2850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:33.536604 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ff2581d7b94dc4883c159a3b478e92. Latest consensus state: current_term: 1 leader_uuid: "30ff2581d7b94dc4883c159a3b478e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } } }
I20260929 14:47:33.536725 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:33.537458 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ff2581d7b94dc4883c159a3b478e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ff2581d7b94dc4883c159a3b478e92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38333 } } }
I20260929 14:47:33.537655 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92 [sys.catalog]: This master's current role is: LEADER
W20260929 14:47:33.540644 2870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:33.540740 2870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:33.540828 2861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:33.541747 2861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:33.547778 2861 catalog_manager.cc:1347] Generated new cluster ID: 9fcebce4d9db43959840fce9f22e3f30
I20260929 14:47:33.547889 2861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:33.561612 2861 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:33.562994 2861 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:33.574843 2861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ff2581d7b94dc4883c159a3b478e92: Generated new TSK 0
I20260929 14:47:33.575738 2861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:33.607889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:33.725427 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:33.725719 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:33.725764 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:33.729954 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:33.730026 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:33.730089 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:33.735762 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:33.737069 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:33.738159 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:33.748165 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:33.748129 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:33.748334 2876 server_base.cc:1034] running on GCE node
W20260929 14:47:33.748153 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:33.748935 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:33.749637 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:33.750824 2876 hybrid_clock.cc:648] HybridClock initialized: now 1790693253750799 us; error 40 us; skew 500 ppm
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:33.753999 2876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:33.755039 2876 webserver.cc:460] Webserver started at http://127.0.98.1:41943/ using document root <none> and password file <none>
I20260929 14:47:33.755791 2876 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:33.755885 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:33.756119 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:33.758389 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1c931d7d2f245c5a845dba647ab2f34"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ddd6dbf0ac19ccc04eaf9d33bb0c9e6e"
server_key_iv: "dbab2122e1d958aa11a31d1480a002e7"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.758991 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c931d7d2f245c5a845dba647ab2f34"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ddd6dbf0ac19ccc04eaf9d33bb0c9e6e"
server_key_iv: "dbab2122e1d958aa11a31d1480a002e7"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.763136 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:47:33.765614 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:33.766803 2876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:33.766953 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "a1c931d7d2f245c5a845dba647ab2f34"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ddd6dbf0ac19ccc04eaf9d33bb0c9e6e"
server_key_iv: "dbab2122e1d958aa11a31d1480a002e7"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.767066 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:33.787099 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:33.790333 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:33.790547 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:33.791133 2876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:33.791965 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:33.792037 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:33.792102 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:33.792150 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:33.800930 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46865
I20260929 14:47:33.800956 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46865 every 8 connection(s)
I20260929 14:47:33.801810 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:33.804785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 2876
I20260929 14:47:33.804904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:33.805227 392 external_mini_cluster.cc:1428] Setting key f7fcf1da8633e6ea6485b7199126b444
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:33.816469 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:33.803824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34807 (local address 127.0.98.62:38333)
0929 14:47:33.803975 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:33.803978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:33.804923 (+ 945us) server_negotiation.cc:407] Connection header received
0929 14:47:33.806511 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:33.806514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:33.806564 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:33.806650 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:33.807829 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.808444 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.809578 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.810006 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.810129 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:33.812771 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:33.812775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:33.812777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:33.812800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:33.814302 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:33.814855 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:33.814859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:33.814860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:33.814968 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:33.815302 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:33.815306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:33.815308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:33.815499 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:33.815606 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:33.816135 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 14:47:33.816293 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:47:33.817286 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:33.804212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.1:34807)
0929 14:47:33.804758 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:33.804818 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:33.806299 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:33.806835 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:33.806842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:33.807193 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:33.807681 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:33.807694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.808575 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.808578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:33.809451 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:33.809460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:33.810165 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:33.810168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:33.810318 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:33.811029 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:33.811057 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:33.812596 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:33.814455 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:33.814462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:33.814477 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:33.814741 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:33.815048 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:33.815052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:33.815054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:33.815196 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:33.815616 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:33.815625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:33.815884 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 14:47:33.816082 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 14:47:33.816450 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:33.818835 3005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260929 14:47:33.819105 3005 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:33.819752 3005 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260929 14:47:33.821398 2814 ts_manager.cc:194] Registered new tserver with Master: a1c931d7d2f245c5a845dba647ab2f34 (127.0.98.1:46865)
I20260929 14:47:33.822641 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:33.835258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:33.946377 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:33.946666 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:33.946740 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:33.950860 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:33.950951 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:33.951056 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:33.956385 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:33.957410 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:33.958432 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:33.968405 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:33.968410 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:33.968405 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:33.968901 3010 server_base.cc:1034] running on GCE node
I20260929 14:47:33.969329 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:33.970094 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:33.971287 3010 hybrid_clock.cc:648] HybridClock initialized: now 1790693253971268 us; error 38 us; skew 500 ppm
Sep 29 14:47:33 dist-test-slave-5hd1 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:33.974555 3010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:33.975580 3010 webserver.cc:460] Webserver started at http://127.0.98.2:34559/ using document root <none> and password file <none>
I20260929 14:47:33.976234 3010 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:33.976325 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:33.976569 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:33.978645 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "806ea58937b34ce0b174bb935af33fae"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ec55efc8970d6568de0cf2749735f4b9"
server_key_iv: "167b31ad8739251ea95e7498d366d5a9"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.979223 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "806ea58937b34ce0b174bb935af33fae"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ec55efc8970d6568de0cf2749735f4b9"
server_key_iv: "167b31ad8739251ea95e7498d366d5a9"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.982952 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:47:33.985546 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:33.986917 3010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:33.987074 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "806ea58937b34ce0b174bb935af33fae"
format_stamp: "Formatted at 2026-09-29 14:47:33 on dist-test-slave-5hd1"
server_key: "ec55efc8970d6568de0cf2749735f4b9"
server_key_iv: "167b31ad8739251ea95e7498d366d5a9"
server_key_version: "encryptionkey@0"
I20260929 14:47:33.987195 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:34.004334 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:34.007709 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:34.007946 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:34.008642 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:34.009531 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:34.009582 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:34.009671 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:34.009718 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:34.018646 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40975
I20260929 14:47:34.018718 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40975 every 8 connection(s)
I20260929 14:47:34.019622 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:34.021879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3010
I20260929 14:47:34.022004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:34.022334 392 external_mini_cluster.cc:1428] Setting key c67fc5e2bd274f42f426d85ebd1fde93
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:47:34.035741 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:34.021488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60279 (local address 127.0.98.62:38333)
0929 14:47:34.021630 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:34.021634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:34.022659 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 14:47:34.024434 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:34.024437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:34.024489 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:34.024576 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:34.026160 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.027030 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.028414 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.028832 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.028971 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:34.031839 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:34.031843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:34.031846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:34.031876 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:34.033405 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:34.033968 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:34.033973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:34.033977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:34.034088 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:34.034551 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:34.034554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:34.034556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:34.034753 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:34.034832 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:34.035445 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 14:47:34.035614 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:47:34.036965 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:34.021904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.2:60279)
0929 14:47:34.022480 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:34.022539 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:34.024164 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:34.024794 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:34.024807 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:34.025373 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:34.026010 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:34.026025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.027177 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.027180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:34.028278 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:34.028289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.029018 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.029022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:34.029184 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:34.029880 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:34.029897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:34.031647 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:34.033576 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:34.033581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:34.033594 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:34.033855 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:34.034184 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:34.034186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:34.034188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:34.034422 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:34.034866 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:34.034903 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:34.035203 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 14:47:34.035363 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 14:47:34.035748 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":153,"threads_started":1}
I20260929 14:47:34.038416 3139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260929 14:47:34.038690 3139 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:34.039201 3139 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260929 14:47:34.040385 2814 ts_manager.cc:194] Registered new tserver with Master: 806ea58937b34ce0b174bb935af33fae (127.0.98.2:40975)
I20260929 14:47:34.040968 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:34.055513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:34.168841 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:34.169127 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:34.169198 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:34.173491 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:34.173576 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:34.173660 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:34.179015 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:34.180101 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:34.181149 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:34.190626 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:34.190626 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:34.190735 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:34.190873 3144 server_base.cc:1034] running on GCE node
I20260929 14:47:34.191438 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:34.192044 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:34.193244 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790693254193210 us; error 53 us; skew 500 ppm
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:34.195921 3144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:34.196822 3144 webserver.cc:460] Webserver started at http://127.0.98.3:40333/ using document root <none> and password file <none>
I20260929 14:47:34.197345 3144 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:34.197403 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:34.197561 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:34.199497 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2b29cd537704c28bfba5502e25ddfa7"
format_stamp: "Formatted at 2026-09-29 14:47:34 on dist-test-slave-5hd1"
server_key: "69f1e6edd446a54195452dc67a340a22"
server_key_iv: "3d571296c5e7087de18a11e2831142a6"
server_key_version: "encryptionkey@0"
I20260929 14:47:34.199992 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2b29cd537704c28bfba5502e25ddfa7"
format_stamp: "Formatted at 2026-09-29 14:47:34 on dist-test-slave-5hd1"
server_key: "69f1e6edd446a54195452dc67a340a22"
server_key_iv: "3d571296c5e7087de18a11e2831142a6"
server_key_version: "encryptionkey@0"
I20260929 14:47:34.203668 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:34.206161 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:34.207479 3144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:34.207583 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "d2b29cd537704c28bfba5502e25ddfa7"
format_stamp: "Formatted at 2026-09-29 14:47:34 on dist-test-slave-5hd1"
server_key: "69f1e6edd446a54195452dc67a340a22"
server_key_iv: "3d571296c5e7087de18a11e2831142a6"
server_key_version: "encryptionkey@0"
I20260929 14:47:34.207727 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:34.246026 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:34.249019 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:34.249225 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:34.249836 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:34.250669 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:34.250741 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:34.250813 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:34.250869 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:34.260957 3144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43053
I20260929 14:47:34.260974 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43053 every 8 connection(s)
I20260929 14:47:34.262003 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:34.262758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3144
I20260929 14:47:34.262907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:34.263213 392 external_mini_cluster.cc:1428] Setting key 43dbccc7fe6c8f6bbf6f07ec501e2008
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:34.277361 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:34.264152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37147 (local address 127.0.98.62:38333)
0929 14:47:34.264361 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:34.264364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:34.265420 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 14:47:34.266904 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:34.266906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:34.266951 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:34.267026 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:34.268263 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.268790 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.270031 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.270246 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.270325 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:34.273146 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:34.273149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:34.273151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:34.273175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:34.274616 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:34.275167 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:34.275170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:34.275171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:34.275250 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:34.275706 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:34.275709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:34.275710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:34.276004 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:34.276117 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:34.277016 (+ 899us) server_negotiation.cc:299] Negotiation successful
0929 14:47:34.277180 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:47:34.278402 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:34.264724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38333 (local address 127.0.98.3:37147)
0929 14:47:34.265226 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:34.265284 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:34.266693 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:34.267196 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:34.267204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:34.267597 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:34.268084 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:34.268097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.268908 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.268911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:34.269920 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:34.269930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:34.270420 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:34.270423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:34.270568 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:34.271362 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:34.271379 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:34.272983 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:34.274731 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:34.274737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:34.274749 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:34.275031 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:34.275396 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:34.275399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:34.275400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:34.275563 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:34.276168 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:34.276175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:34.276511 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 14:47:34.276883 (+ 372us) client_negotiation.cc:240] Negotiation successful
0929 14:47:34.277242 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":106,"threads_started":1}
I20260929 14:47:34.279716 3273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38333
I20260929 14:47:34.279982 3273 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:34.280594 3273 heartbeater.cc:511] Master 127.0.98.62:38333 requested a full tablet report, sending...
I20260929 14:47:34.281742 2814 ts_manager.cc:194] Registered new tserver with Master: d2b29cd537704c28bfba5502e25ddfa7 (127.0.98.3:43053)
I20260929 14:47:34.282380 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37147
I20260929 14:47:34.289949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:47:34.303413 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:34.301872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.0.98.62:38333)
0929 14:47:34.302043 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:34.302048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:34.302184 (+ 136us) server_negotiation.cc:407] Connection header received
0929 14:47:34.302308 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:34.302310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:34.302353 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:34.302424 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:34.303251 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59506: BlockingRecv error: recv got EOF from 127.0.0.1:59506 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:47:34.304455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2876
I20260929 14:47:34.310420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3010
I20260929 14:47:34.315913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3144
I20260929 14:47:34.320878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 2782
2026-09-29T14:47:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2194 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[3285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[3285](info): set up 2 sockets
Sep 29 14:47:34 dist-test-slave-5hd1 krb5kdc[3285](info): commencing operation
krb5kdc: starting...
W20260929 14:47:35.368083 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:35.493441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:35.605028 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:35.605289 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:35.605330 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:35.609640 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:35.609696 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:35.609716 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:35.609733 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:35.609748 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:35.615000 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:35.615998 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:35.617017 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:35.625372 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:35.625442 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:35.625384 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:35.625563 3298 server_base.cc:1034] running on GCE node
I20260929 14:47:35.626202 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:35.627296 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:35.628472 3298 hybrid_clock.cc:648] HybridClock initialized: now 1790693255628449 us; error 48 us; skew 500 ppm
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:35.639905 3298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:35.641036 3298 webserver.cc:460] Webserver started at http://127.0.98.62:40179/ using document root <none> and password file <none>
I20260929 14:47:35.641619 3298 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:35.641713 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:35.641952 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:35.644085 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "68e14a6060a3490fbb3ac1f2e7486854"
format_stamp: "Formatted at 2026-09-29 14:47:35 on dist-test-slave-5hd1"
server_key: "1138b61635b0561179582f11f58527e2"
server_key_iv: "a80cf02d78c43162f2066f1fd9055994"
server_key_version: "encryptionkey@0"
I20260929 14:47:35.644711 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "68e14a6060a3490fbb3ac1f2e7486854"
format_stamp: "Formatted at 2026-09-29 14:47:35 on dist-test-slave-5hd1"
server_key: "1138b61635b0561179582f11f58527e2"
server_key_iv: "a80cf02d78c43162f2066f1fd9055994"
server_key_version: "encryptionkey@0"
I20260929 14:47:35.648402 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:47:35.650774 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:35.652109 3298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:35.652285 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "68e14a6060a3490fbb3ac1f2e7486854"
format_stamp: "Formatted at 2026-09-29 14:47:35 on dist-test-slave-5hd1"
server_key: "1138b61635b0561179582f11f58527e2"
server_key_iv: "a80cf02d78c43162f2066f1fd9055994"
server_key_version: "encryptionkey@0"
I20260929 14:47:35.652402 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:35.666127 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:35.668927 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:35.669127 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:35.675936 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36149
I20260929 14:47:35.676270 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36149 every 8 connection(s)
I20260929 14:47:35.676930 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:35.679768 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:35.679790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3298
I20260929 14:47:35.679898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:35.680219 392 external_mini_cluster.cc:1428] Setting key 3b129c3c1f9a7c3b5372053bdfaf0dc8
I20260929 14:47:35.686641 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: Bootstrap starting.
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:35.689292 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:35.690199 3366 log.cc:824] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:35.692315 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: No bootstrap required, opened a new log
I20260929 14:47:35.694371 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:35.681461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.0.98.62:36149)
0929 14:47:35.681875 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:35.681889 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:35.681922 (+ 33us) server_negotiation.cc:407] Connection header received
0929 14:47:35.682546 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:35.682562 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:35.682867 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:35.683159 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:35.684061 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:35.685116 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:35.686236 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:35.686686 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:35.686869 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:35.689536 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:35.689542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:35.689562 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:35.689595 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:35.691411 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:35.691832 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:35.691838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:35.691844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:35.691985 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:35.692279 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:35.692282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:35.692284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:35.692765 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:35.692961 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:35.693147 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 14:47:35.693447 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":133,"threads_started":1}
I20260929 14:47:35.696718 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } }
I20260929 14:47:35.696925 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:35.696981 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e14a6060a3490fbb3ac1f2e7486854, State: Initialized, Role: FOLLOWER
I20260929 14:47:35.697368 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } }
I20260929 14:47:35.697544 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:47:35.697570 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:35.697636 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:35.697753 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:35.698613 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } }
I20260929 14:47:35.698938 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68e14a6060a3490fbb3ac1f2e7486854; no voters:
I20260929 14:47:35.699255 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:35.699483 3372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:35.699734 3372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [term 1 LEADER]: Becoming Leader. State: Replica: 68e14a6060a3490fbb3ac1f2e7486854, State: Running, Role: LEADER
I20260929 14:47:35.700057 3372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } }
I20260929 14:47:35.700691 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:35.701756 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68e14a6060a3490fbb3ac1f2e7486854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } } }
I20260929 14:47:35.701890 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:35.702119 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68e14a6060a3490fbb3ac1f2e7486854. Latest consensus state: current_term: 1 leader_uuid: "68e14a6060a3490fbb3ac1f2e7486854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e14a6060a3490fbb3ac1f2e7486854" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36149 } } }
I20260929 14:47:35.702210 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854 [sys.catalog]: This master's current role is: LEADER
W20260929 14:47:35.705667 3387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:35.705737 3387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:35.705822 3381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:35.706517 3381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:35.711952 3381 catalog_manager.cc:1347] Generated new cluster ID: 481b6ad8f81348e0929cdd7096e5cc1d
I20260929 14:47:35.712019 3381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:35.720007 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:35.714445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.0.98.62:36149)
0929 14:47:35.714547 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:35.714552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:35.714810 (+ 258us) server_negotiation.cc:407] Connection header received
0929 14:47:35.714927 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:35.714931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:35.714985 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:35.715080 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:35.715643 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:35.716515 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:35.717614 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:35.717973 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:35.718083 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:35.718983 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:35.718987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:35.718990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:35.719559 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:35.719592 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:35.719611 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:35.719699 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:35.719800 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:47:35.719860 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:47:35.737839 3381 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:35.738945 3381 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:35.745968 3381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68e14a6060a3490fbb3ac1f2e7486854: Generated new TSK 0
I20260929 14:47:35.746742 3381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:47:35.755923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3298
2026-09-29T14:47:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1432 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3399](info): set up 2 sockets
Sep 29 14:47:35 dist-test-slave-5hd1 krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260929 14:47:36.792150 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:36 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:36.925901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:37.051471 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:37.051892 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:37.052117 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:37.056742 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:37.056874 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:37.056910 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:37.056948 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:37.057000 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:37.063333 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:37.064718 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:37.065858 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:37.075457 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.075459 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.075459 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:37.075502 3412 server_base.cc:1034] running on GCE node
I20260929 14:47:37.076541 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:37.078076 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:37.079479 3412 hybrid_clock.cc:648] HybridClock initialized: now 1790693257079391 us; error 41 us; skew 500 ppm
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:37.083643 3412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:37.084875 3412 webserver.cc:460] Webserver started at http://127.0.98.62:33613/ using document root <none> and password file <none>
I20260929 14:47:37.085552 3412 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:37.085665 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:37.085938 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:37.088363 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "a8c1100f18fd4be39d0ecca50a7fc313"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "7e29311c89bd3732a3ea294067980923"
server_key_iv: "268beb6263ef59315a74e00a1116fee7"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.089037 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8c1100f18fd4be39d0ecca50a7fc313"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "7e29311c89bd3732a3ea294067980923"
server_key_iv: "268beb6263ef59315a74e00a1116fee7"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.093319 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:37.096002 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.097533 3412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:37.097704 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "a8c1100f18fd4be39d0ecca50a7fc313"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "7e29311c89bd3732a3ea294067980923"
server_key_iv: "268beb6263ef59315a74e00a1116fee7"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.097839 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:37.119318 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:37.122985 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:37.123179 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:37.131310 3412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36725
I20260929 14:47:37.131614 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36725 every 8 connection(s)
I20260929 14:47:37.132591 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:37.134131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3412
I20260929 14:47:37.134229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:37.134550 392 external_mini_cluster.cc:1428] Setting key 54031b36a3971d1889c0036a4db22309
I20260929 14:47:37.135876 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:37.141872 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313: Bootstrap starting.
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:37.145712 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:37.146570 3480 log.cc:824] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:37.149077 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313: No bootstrap required, opened a new log
I20260929 14:47:37.151085 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.136067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.62:36725)
0929 14:47:37.136579 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:37.136607 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:37.136644 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:47:37.137301 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:37.137318 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:37.137657 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:37.138029 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:37.138882 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.139898 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.141309 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.141834 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.142020 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.145594 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:37.145601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:37.145617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:37.145653 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:37.147567 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.148122 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.148127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.148132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.148324 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.148703 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.148706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.148707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.149143 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:37.149380 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:37.149766 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 14:47:37.150121 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":134,"threads_started":1}
I20260929 14:47:37.152750 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260929 14:47:37.153036 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:37.153086 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c1100f18fd4be39d0ecca50a7fc313, State: Initialized, Role: FOLLOWER
I20260929 14:47:37.153571 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260929 14:47:37.153703 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:37.153745 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:37.153823 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:37.154779 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260929 14:47:37.155136 3480 leader_election.cc:302] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c1100f18fd4be39d0ecca50a7fc313; no voters:
I20260929 14:47:37.155426 3480 leader_election.cc:288] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:37.155728 3485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:37.156246 3485 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [term 1 LEADER]: Becoming Leader. State: Replica: a8c1100f18fd4be39d0ecca50a7fc313, State: Running, Role: LEADER
I20260929 14:47:37.156889 3485 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260929 14:47:37.156977 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:37.158802 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } } }
I20260929 14:47:37.158962 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:37.159214 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c1100f18fd4be39d0ecca50a7fc313. Latest consensus state: current_term: 1 leader_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c1100f18fd4be39d0ecca50a7fc313" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } } }
I20260929 14:47:37.159710 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:37.159315 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:37.162436 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:37.168804 3499 catalog_manager.cc:1347] Generated new cluster ID: 1d761d38acee4e7082352e835d8f9aee
I20260929 14:47:37.168891 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:37.176236 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:37.177944 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:37.184072 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c1100f18fd4be39d0ecca50a7fc313: Generated new TSK 0
I20260929 14:47:37.184912 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:37.216384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:37.329718 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:37.329962 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:37.330001 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:37.334021 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:37.334070 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:37.334125 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:37.339663 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:37.340665 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:37.341651 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:37.350906 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.350975 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.350936 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:37.351259 3506 server_base.cc:1034] running on GCE node
I20260929 14:47:37.351652 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:37.352303 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:37.353480 3506 hybrid_clock.cc:648] HybridClock initialized: now 1790693257353469 us; error 29 us; skew 500 ppm
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:37.356024 3506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:37.356889 3506 webserver.cc:460] Webserver started at http://127.0.98.1:42453/ using document root <none> and password file <none>
I20260929 14:47:37.357457 3506 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:37.357518 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:37.357714 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:37.359700 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d78d0df714c4d1c8377f4239d319f52"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "50f42a23f3de88e3a4cb049f47e8e9ab"
server_key_iv: "3777df9d0c3684a7c4870797a6e25ad5"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.360224 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d78d0df714c4d1c8377f4239d319f52"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "50f42a23f3de88e3a4cb049f47e8e9ab"
server_key_iv: "3777df9d0c3684a7c4870797a6e25ad5"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.363602 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:37.365854 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.367066 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:37.367177 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "0d78d0df714c4d1c8377f4239d319f52"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "50f42a23f3de88e3a4cb049f47e8e9ab"
server_key_iv: "3777df9d0c3684a7c4870797a6e25ad5"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.367273 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:37.379284 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:37.382359 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:37.382562 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:37.383162 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:37.383965 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:37.384011 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.384080 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:37.384110 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.392974 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40391
I20260929 14:47:37.392993 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40391 every 8 connection(s)
I20260929 14:47:37.393839 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:37.402662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3506
I20260929 14:47:37.402796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:37.403126 392 external_mini_cluster.cc:1428] Setting key 7ade0009d9f4a2c98ee12eb56dc2c381
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:37.408627 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.395633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33209 (local address 127.0.98.62:36725)
0929 14:47:37.395776 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:37.395779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:37.396644 (+ 865us) server_negotiation.cc:407] Connection header received
0929 14:47:37.397941 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:37.397943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:37.397988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:37.398061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:37.399230 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.399792 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.401262 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.401525 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.401620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.404662 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:37.404665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:37.404667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:37.404688 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:37.406151 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.406796 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.406799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.406801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.406893 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.407286 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.407292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.407297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.407586 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:37.407693 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:37.408234 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 14:47:37.408398 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:47:37.409394 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.395991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.1:33209)
0929 14:47:37.396496 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:37.396536 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:37.397755 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:37.398218 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:37.398224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:37.398610 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:37.399078 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.399095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.399910 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.399913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.401150 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.401160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.401668 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.401671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.401808 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.402546 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:37.402568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:37.404521 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:37.406274 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.406283 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.406300 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.406676 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.406984 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.406989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.406992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.407176 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.407691 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:37.407696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:37.407923 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 14:47:37.408078 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 14:47:37.408399 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":172,"threads_started":1}
I20260929 14:47:37.410658 3635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260929 14:47:37.410883 3635 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:37.411355 3635 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:37.412953 3443 ts_manager.cc:194] Registered new tserver with Master: 0d78d0df714c4d1c8377f4239d319f52 (127.0.98.1:40391)
I20260929 14:47:37.414055 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:37.432276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:37.541108 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:37.541389 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:37.541432 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:37.545498 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:37.545567 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:37.545626 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:37.551034 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:37.552143 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:37.553190 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:37.562657 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:37.562789 3640 server_base.cc:1034] running on GCE node
W20260929 14:47:37.562667 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.562709 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:37.563261 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:37.563827 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:37.565039 3640 hybrid_clock.cc:648] HybridClock initialized: now 1790693257564985 us; error 46 us; skew 500 ppm
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:37.568743 3640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:37.569859 3640 webserver.cc:460] Webserver started at http://127.0.98.2:44119/ using document root <none> and password file <none>
I20260929 14:47:37.570567 3640 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:37.570641 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:37.570976 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:37.573503 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "923bd0080b1549f78635cd882a1a5f4d"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "2c28cbecc55ab5ee6a256e24502294ec"
server_key_iv: "27905273eb27e568a4e4c17d3558b323"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.574206 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "923bd0080b1549f78635cd882a1a5f4d"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "2c28cbecc55ab5ee6a256e24502294ec"
server_key_iv: "27905273eb27e568a4e4c17d3558b323"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.578994 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 14:47:37.582170 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.583799 3640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:37.583966 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "923bd0080b1549f78635cd882a1a5f4d"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "2c28cbecc55ab5ee6a256e24502294ec"
server_key_iv: "27905273eb27e568a4e4c17d3558b323"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.584103 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:37.618518 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:37.622004 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:37.622233 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:37.622936 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:37.623917 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:37.623991 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.624063 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:37.624121 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.635183 3640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33839
I20260929 14:47:37.635206 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33839 every 8 connection(s)
I20260929 14:47:37.636343 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:37.638903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3640
I20260929 14:47:37.639041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:37.639411 392 external_mini_cluster.cc:1428] Setting key 0602e1c6ef709fc4400f440e7a08bec6
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:47:37.654275 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.638515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38381 (local address 127.0.98.62:36725)
0929 14:47:37.638732 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:37.638738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:37.639942 (+ 1204us) server_negotiation.cc:407] Connection header received
0929 14:47:37.642004 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:37.642007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:37.642056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:37.642159 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:37.643981 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.644676 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.645962 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.646442 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.646617 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.649494 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:37.649501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:37.649507 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:37.649547 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:37.651398 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.652093 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.652096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.652098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.652244 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.652586 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.652589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.652590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.652776 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:37.652916 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:37.653883 (+ 967us) server_negotiation.cc:299] Negotiation successful
0929 14:47:37.654045 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:47:37.655229 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.639050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.2:38381)
0929 14:47:37.639720 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:37.639786 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:37.641699 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:37.642428 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:37.642442 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:37.643077 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:37.643823 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.643839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.644797 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.644801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.645762 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.645772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.646635 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.646638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.646792 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.647616 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:37.647644 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:37.649267 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:37.651531 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.651536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.651551 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.651976 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.652344 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.652346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.652348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.652485 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.653025 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:37.653032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:37.653520 (+ 488us) client_negotiation.cc:769] Sending connection context
0929 14:47:37.653803 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 14:47:37.654177 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":165,"threads_started":1}
I20260929 14:47:37.656708 3769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260929 14:47:37.657114 3769 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:37.657856 3769 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
I20260929 14:47:37.659337 3443 ts_manager.cc:194] Registered new tserver with Master: 923bd0080b1549f78635cd882a1a5f4d (127.0.98.2:33839)
I20260929 14:47:37.660298 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:37.671058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:37.781903 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:37.782177 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:37.782219 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:37.786207 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:37.786266 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:37.786365 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:37.791723 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:37.792778 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:37.793792 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:37.803679 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.803697 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:37.803697 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:37.803936 3774 server_base.cc:1034] running on GCE node
I20260929 14:47:37.804597 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:37.805354 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:37.806556 3774 hybrid_clock.cc:648] HybridClock initialized: now 1790693257806521 us; error 48 us; skew 500 ppm
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:37.809820 3774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:37.810870 3774 webserver.cc:460] Webserver started at http://127.0.98.3:43069/ using document root <none> and password file <none>
I20260929 14:47:37.811533 3774 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:37.811622 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:37.811872 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:37.814415 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "509a194c02bd43cb95646c2ff2cc9d6b"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "9779ae2c4d5f46360c008e908407aad6"
server_key_iv: "b34b02be2d3af528c2ab6f241f28c3cb"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.815037 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "509a194c02bd43cb95646c2ff2cc9d6b"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "9779ae2c4d5f46360c008e908407aad6"
server_key_iv: "b34b02be2d3af528c2ab6f241f28c3cb"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.819437 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:47:37.822540 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.824297 3774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:37.824461 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "509a194c02bd43cb95646c2ff2cc9d6b"
format_stamp: "Formatted at 2026-09-29 14:47:37 on dist-test-slave-5hd1"
server_key: "9779ae2c4d5f46360c008e908407aad6"
server_key_iv: "b34b02be2d3af528c2ab6f241f28c3cb"
server_key_version: "encryptionkey@0"
I20260929 14:47:37.824606 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:37.850070 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:37.853968 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:37.854211 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:37.855005 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:37.856030 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:37.856109 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.856220 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:37.856276 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:37.868391 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46755
I20260929 14:47:37.868515 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46755 every 8 connection(s)
I20260929 14:47:37.869532 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:37.878813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3774
I20260929 14:47:37.878926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:37.879294 392 external_mini_cluster.cc:1428] Setting key bd53840667756c1c262aa4baae2d80fc
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:37.885566 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.871752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60377 (local address 127.0.98.62:36725)
0929 14:47:37.871937 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:37.871942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:37.873008 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 14:47:37.874633 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:37.874636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:37.874689 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:37.874786 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:37.876440 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.876975 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.878248 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.878457 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.878581 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.881842 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:37.881847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:37.881849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:37.881879 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:37.883300 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.883864 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.883870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.883876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.883951 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.884391 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.884395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.884397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.884604 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:37.884716 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:37.885253 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 14:47:37.885387 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:47:37.886570 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.872212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.3:60377)
0929 14:47:37.872799 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:37.872852 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:37.874395 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:37.875030 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:37.875038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:37.875463 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:37.876274 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.876302 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.877094 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.877096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.878135 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:37.878144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.878617 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.878619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:37.878795 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.879614 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:37.879632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:37.881633 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:37.883438 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.883444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.883461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.883748 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.884058 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:37.884061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:37.884063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:37.884289 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:37.884712 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:37.884719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:37.885002 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 14:47:37.885202 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 14:47:37.885563 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":155,"threads_started":1}
I20260929 14:47:37.887934 3903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260929 14:47:37.888279 3903 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:37.888936 3903 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
I20260929 14:47:37.890049 3443 ts_manager.cc:194] Registered new tserver with Master: 509a194c02bd43cb95646c2ff2cc9d6b (127.0.98.3:46755)
I20260929 14:47:37.890676 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60377
I20260929 14:47:37.893653 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 14:47:37 dist-test-slave-5hd1 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693257, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:37.922078 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:37.912645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.98.62:36725)
0929 14:47:37.912869 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:37.912875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:37.913048 (+ 173us) server_negotiation.cc:407] Connection header received
0929 14:47:37.913263 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:37.913266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:37.913315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:37.913411 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:37.914192 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.914774 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.915659 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:37.915889 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:37.916007 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:37.918930 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:37.918933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:37.918935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:37.918958 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:37.920349 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.920777 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.920779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.920781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.920850 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:37.921194 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:37.921196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:37.921197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:37.921433 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:37.921544 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:37.921825 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 14:47:37.921927 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 14:47:37.922610 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33966
I20260929 14:47:37.923727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3506
I20260929 14:47:37.929988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3640
I20260929 14:47:37.935794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3774
I20260929 14:47:37.942233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3412
2026-09-29T14:47:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2190 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:38 dist-test-slave-5hd1 krb5kdc[3915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:38 dist-test-slave-5hd1 krb5kdc[3915](info): set up 2 sockets
Sep 29 14:47:38 dist-test-slave-5hd1 krb5kdc[3915](info): commencing operation
krb5kdc: starting...
W20260929 14:47:39.032509 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:39.169239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38143
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:39.282583 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:39.282883 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:39.282976 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:39.287222 3928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:39.287307 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:39.287334 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:39.287351 3928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:39.287367 3928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:39.292964 3928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:39.294088 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:39.295131 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:39.305142 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:39.305154 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:39.305121 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:39.305198 3928 server_base.cc:1034] running on GCE node
I20260929 14:47:39.306270 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:39.307777 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:39.308995 3928 hybrid_clock.cc:648] HybridClock initialized: now 1790693259308976 us; error 43 us; skew 500 ppm
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:39.312491 3928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:39.313650 3928 webserver.cc:460] Webserver started at http://127.0.98.62:44779/ using document root <none> and password file <none>
I20260929 14:47:39.314356 3928 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:39.314453 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:39.314742 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:39.317381 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "d381b06f6edb4c3ab792edb66305ffd8"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "3cd464956080f8587e6ec6022bd7473a"
server_key_iv: "ba4b05a915b59efb3ef8e71e7b0321ea"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.318029 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "d381b06f6edb4c3ab792edb66305ffd8"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "3cd464956080f8587e6ec6022bd7473a"
server_key_iv: "ba4b05a915b59efb3ef8e71e7b0321ea"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.322643 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:47:39.325857 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.327651 3928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:39.327934 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "d381b06f6edb4c3ab792edb66305ffd8"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "3cd464956080f8587e6ec6022bd7473a"
server_key_iv: "ba4b05a915b59efb3ef8e71e7b0321ea"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.328089 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:39.353266 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:39.357430 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:39.357689 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:39.367916 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38143
I20260929 14:47:39.367961 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38143 every 8 connection(s)
I20260929 14:47:39.369185 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:39.372563 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:39.377422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 3928
I20260929 14:47:39.377610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:39.377945 392 external_mini_cluster.cc:1428] Setting key 16fe4ebf4aaad2725444ec2801fd6d10
I20260929 14:47:39.378510 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: Bootstrap starting.
I20260929 14:47:39.381309 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:39.382097 3996 log.cc:824] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:39.384302 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: No bootstrap required, opened a new log
I20260929 14:47:39.387982 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } }
I20260929 14:47:39.388295 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:39.388404 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d381b06f6edb4c3ab792edb66305ffd8, State: Initialized, Role: FOLLOWER
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:39.388940 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } }
I20260929 14:47:39.389072 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:39.389158 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:39.389283 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:39.390440 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } }
I20260929 14:47:39.390779 3996 leader_election.cc:302] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d381b06f6edb4c3ab792edb66305ffd8; no voters:
I20260929 14:47:39.391106 3996 leader_election.cc:288] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:39.391278 4001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:39.391607 4001 raft_consensus.cc:686] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [term 1 LEADER]: Becoming Leader. State: Replica: d381b06f6edb4c3ab792edb66305ffd8, State: Running, Role: LEADER
I20260929 14:47:39.392071 4001 consensus_queue.cc:237] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } }
I20260929 14:47:39.392552 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:39.394207 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d381b06f6edb4c3ab792edb66305ffd8. Latest consensus state: current_term: 1 leader_uuid: "d381b06f6edb4c3ab792edb66305ffd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } } }
I20260929 14:47:39.394351 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:39.395973 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d381b06f6edb4c3ab792edb66305ffd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381b06f6edb4c3ab792edb66305ffd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38143 } } }
I20260929 14:47:39.396093 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:39.396095 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:39.379684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.62:38143)
0929 14:47:39.380357 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:39.380379 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:39.380428 (+ 49us) server_negotiation.cc:407] Connection header received
0929 14:47:39.381287 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:39.381309 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:39.381828 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:39.382249 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:39.383120 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.384427 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.385346 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.385892 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.386075 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:39.389293 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:39.389299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:39.389315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:39.389347 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:39.391676 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.392409 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.392416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.392423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.392577 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.392949 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.392956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.392959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.393588 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:39.393815 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:39.394688 (+ 873us) server_negotiation.cc:299] Negotiation successful
0929 14:47:39.395063 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":180,"threads_started":1}
W20260929 14:47:39.397997 4016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:39.398088 4016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:39.398350 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:39.399228 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:39.405990 4013 catalog_manager.cc:1347] Generated new cluster ID: 884a565594704d1da64e8ba4efee710b
I20260929 14:47:39.406072 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:39.414989 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:39.415956 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:39.423978 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d381b06f6edb4c3ab792edb66305ffd8: Generated new TSK 0
I20260929 14:47:39.424782 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:39.460847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:39.600073 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:39.600440 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:39.600489 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:39.605165 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:39.605257 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:39.605329 4022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:39.611675 4022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:39.613243 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:39.614418 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:47:39.625612 4022 server_base.cc:1034] running on GCE node
W20260929 14:47:39.625552 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:39.625581 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:39.625581 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:39.626646 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:39.627446 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:39.628657 4022 hybrid_clock.cc:648] HybridClock initialized: now 1790693259628631 us; error 51 us; skew 500 ppm
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:39.631995 4022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:39.633208 4022 webserver.cc:460] Webserver started at http://127.0.98.1:36791/ using document root <none> and password file <none>
I20260929 14:47:39.633926 4022 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:39.634027 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:39.634312 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:39.636905 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "8820ebee151841e6a7314d281a91ef29"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "7bb50ec34707eb7853b154ade4d9255c"
server_key_iv: "69c17ee44ac1fa71b8952358816032e2"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.637573 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8820ebee151841e6a7314d281a91ef29"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "7bb50ec34707eb7853b154ade4d9255c"
server_key_iv: "69c17ee44ac1fa71b8952358816032e2"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.642371 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:47:39.645612 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.647416 4022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:39.647603 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "8820ebee151841e6a7314d281a91ef29"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "7bb50ec34707eb7853b154ade4d9255c"
server_key_iv: "69c17ee44ac1fa71b8952358816032e2"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.647759 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:39.665042 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:39.669338 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:39.669591 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:39.670380 4022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:39.671561 4022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:39.671649 4022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.671727 4022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:39.671772 4022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.683930 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44349
I20260929 14:47:39.684044 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44349 every 8 connection(s)
I20260929 14:47:39.685308 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:39.689613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4022
I20260929 14:47:39.689728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:39.690071 392 external_mini_cluster.cc:1428] Setting key 519f24e96d2dc152799b7e87cef30f76
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:39.703660 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:39.687761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60815 (local address 127.0.98.62:38143)
0929 14:47:39.687939 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:39.687943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:39.689030 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 14:47:39.690814 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:39.690819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:39.690886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:39.691015 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:39.692671 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.693634 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.695151 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.695425 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.695552 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:39.698886 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:39.698894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:39.698900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:39.698937 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:39.700764 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.701427 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.701431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.701433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.701729 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.702147 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.702152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.702156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.702487 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:39.702616 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:39.703345 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 14:47:39.703469 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 14:47:39.704847 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:39.688236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38143 (local address 127.0.98.1:60815)
0929 14:47:39.688820 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:39.688877 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:39.690539 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:39.691245 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:39.691255 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:39.691782 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:39.692502 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:39.692518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.693797 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.693801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:39.694995 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:39.695006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.695597 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.695601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:39.695832 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:39.696611 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:39.696638 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:39.698647 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:39.700922 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:39.700928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:39.700946 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:39.701283 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:39.701858 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:39.701861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:39.701862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:39.702010 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:39.702633 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:39.702641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:39.702994 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 14:47:39.703212 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 14:47:39.703600 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:39.706369 4151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38143
I20260929 14:47:39.706681 4151 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:39.707283 4151 heartbeater.cc:511] Master 127.0.98.62:38143 requested a full tablet report, sending...
I20260929 14:47:39.708985 3960 ts_manager.cc:194] Registered new tserver with Master: 8820ebee151841e6a7314d281a91ef29 (127.0.98.1:44349)
I20260929 14:47:39.710295 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:39.723335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:39.869994 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:39.870443 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:39.870540 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:39.875330 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:39.875466 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:39.875577 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:39.882769 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:39.884514 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:39.886020 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:39.898309 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:39.898330 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:39.898679 4156 server_base.cc:1034] running on GCE node
W20260929 14:47:39.898370 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:39.899375 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:39.900400 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:39.901657 4156 hybrid_clock.cc:648] HybridClock initialized: now 1790693259901624 us; error 68 us; skew 500 ppm
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:39.905315 4156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:39.906662 4156 webserver.cc:460] Webserver started at http://127.0.98.2:33809/ using document root <none> and password file <none>
I20260929 14:47:39.907410 4156 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:39.907517 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:39.907799 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:39.910413 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "7bb85162b135478084f6cd7622432055"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "5ad933a4cebdc63ce31e2aa50107d0a0"
server_key_iv: "c8e73ff98a29cd81a8aa7dca13f93948"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.911142 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb85162b135478084f6cd7622432055"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "5ad933a4cebdc63ce31e2aa50107d0a0"
server_key_iv: "c8e73ff98a29cd81a8aa7dca13f93948"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.916121 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 14:47:39.919691 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.921875 4156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:47:39.922081 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "7bb85162b135478084f6cd7622432055"
format_stamp: "Formatted at 2026-09-29 14:47:39 on dist-test-slave-5hd1"
server_key: "5ad933a4cebdc63ce31e2aa50107d0a0"
server_key_iv: "c8e73ff98a29cd81a8aa7dca13f93948"
server_key_version: "encryptionkey@0"
I20260929 14:47:39.922231 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:39.936414 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:39.940353 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:39.940598 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:39.941385 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:39.942437 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:39.942520 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.942603 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:39.942659 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:39.953589 4156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46195
I20260929 14:47:39.953621 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46195 every 8 connection(s)
I20260929 14:47:39.954632 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:39.962297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4156
I20260929 14:47:39.962452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:39.962901 392 external_mini_cluster.cc:1428] Setting key 70f3198ee497ec16c934008f2b2dfa8a
Sep 29 14:47:39 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:39.972288 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:39.957397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50291 (local address 127.0.98.62:38143)
0929 14:47:39.957540 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:39.957543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:39.958706 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 14:47:39.960241 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:39.960245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:39.960300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:39.960410 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:39.961656 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.962180 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.963481 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.963720 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.963825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:39.967591 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:39.967595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:39.967597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:39.967629 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:39.969393 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.970166 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.970178 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.970179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.970255 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:39.970778 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:39.970781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:39.970783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:39.970990 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:39.971082 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:39.971989 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 14:47:39.972093 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:47:39.974227 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:39.957905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38143 (local address 127.0.98.2:50291)
0929 14:47:39.958521 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:39.958576 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:39.959973 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:39.960612 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:39.960620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:39.961005 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:39.961517 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:39.961529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.962302 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.962306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:39.963350 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:39.963359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:39.963962 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:39.963970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:39.964293 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:39.965168 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:39.965188 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:39.967311 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:39.969541 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:39.969552 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:39.969577 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:39.970042 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:39.970369 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:39.970376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:39.970382 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:39.970619 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:39.971122 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:39.971133 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:39.971632 (+ 499us) client_negotiation.cc:769] Sending connection context
0929 14:47:39.971961 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 14:47:39.972544 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":168,"threads_started":1}
I20260929 14:47:39.976034 4285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38143
I20260929 14:47:39.976464 4285 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:47:39.977322 4285 heartbeater.cc:511] Master 127.0.98.62:38143 requested a full tablet report, sending...
I20260929 14:47:39.978857 3960 ts_manager.cc:194] Registered new tserver with Master: 7bb85162b135478084f6cd7622432055 (127.0.98.2:46195)
I20260929 14:47:39.979530 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:47:40.712975 4151 heartbeater.cc:503] Master 127.0.98.62:38143 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 14:47:40.753026 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.788s user 0.002s sys 0.002s
I20260929 14:47:40.753738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:40.902244 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:40.902531 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:40.902618 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:40.906965 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:40.907038 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:40.907099 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:40.912580 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:40.913857 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:40.915186 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:40.926388 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:40.926393 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:40.926393 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:40.926995 4290 server_base.cc:1034] running on GCE node
I20260929 14:47:40.927476 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:40.928149 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:40.929494 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790693260929464 us; error 40 us; skew 500 ppm
Sep 29 14:47:40 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:40.938807 4290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:40.940466 4290 webserver.cc:460] Webserver started at http://127.0.98.3:41513/ using document root <none> and password file <none>
I20260929 14:47:40.941179 4290 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:40.941249 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:40.941452 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:40.943940 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "edb846c09b054bcc96f54eada8cea7f2"
format_stamp: "Formatted at 2026-09-29 14:47:40 on dist-test-slave-5hd1"
server_key: "ba0ed94d9b992488756b6494754ba0fd"
server_key_iv: "641725d1d62640d4d74c6f970f3d77c2"
server_key_version: "encryptionkey@0"
I20260929 14:47:40.944672 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "edb846c09b054bcc96f54eada8cea7f2"
format_stamp: "Formatted at 2026-09-29 14:47:40 on dist-test-slave-5hd1"
server_key: "ba0ed94d9b992488756b6494754ba0fd"
server_key_iv: "641725d1d62640d4d74c6f970f3d77c2"
server_key_version: "encryptionkey@0"
I20260929 14:47:40.949433 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:40.952831 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:40.954880 4290 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:47:40.955046 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "edb846c09b054bcc96f54eada8cea7f2"
format_stamp: "Formatted at 2026-09-29 14:47:40 on dist-test-slave-5hd1"
server_key: "ba0ed94d9b992488756b6494754ba0fd"
server_key_iv: "641725d1d62640d4d74c6f970f3d77c2"
server_key_version: "encryptionkey@0"
I20260929 14:47:40.955204 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:40.982193 4285 heartbeater.cc:503] Master 127.0.98.62:38143 was elected leader, sending a full tablet report...
I20260929 14:47:40.984151 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:40.988283 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:40.988526 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:40.989317 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:40.990391 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:40.990471 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:40.990561 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:40.990608 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:41.004431 4290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38259
I20260929 14:47:41.004482 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38259 every 8 connection(s)
I20260929 14:47:41.005678 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:41.013732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4290
I20260929 14:47:41.013851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:41.014170 392 external_mini_cluster.cc:1428] Setting key 9024f367b1b30ea25f414ebe5f618ad7
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:41.022692 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:41.007978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58863 (local address 127.0.98.62:38143)
0929 14:47:41.008437 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:41.008441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:41.009183 (+ 742us) server_negotiation.cc:407] Connection header received
0929 14:47:41.010849 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:41.010853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:41.010925 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:41.011061 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:41.012594 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.013196 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.014640 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.014947 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.015069 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:41.018493 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:41.018496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:41.018499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:41.018527 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:41.020135 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.020798 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.020801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.020802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.020906 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.021433 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.021435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.021437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.021639 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:41.021722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:41.022413 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 14:47:41.022548 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":191,"threads_started":1}
I20260929 14:47:41.023859 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:41.008454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38143 (local address 127.0.98.3:58863)
0929 14:47:41.008985 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:41.009038 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:41.010572 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:41.011292 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:41.011301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:41.011750 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:41.012408 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:41.012426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.013333 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.013337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:41.014458 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:41.014469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.015123 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.015128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:41.015383 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:41.016267 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:41.016289 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:41.018297 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:41.020331 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:41.020336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:41.020353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:41.020660 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:41.020997 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:41.020999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:41.021000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:41.021147 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:41.021737 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:41.021744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:41.022069 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 14:47:41.022264 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 14:47:41.022618 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260929 14:47:41.025665 4419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38143
I20260929 14:47:41.025993 4419 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:41.026569 4419 heartbeater.cc:511] Master 127.0.98.62:38143 requested a full tablet report, sending...
I20260929 14:47:41.027760 3960 ts_manager.cc:194] Registered new tserver with Master: edb846c09b054bcc96f54eada8cea7f2 (127.0.98.3:38259)
I20260929 14:47:41.028512 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58863
I20260929 14:47:41.029393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 14:47:41.134862 392 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.105s user 0.002s sys 0.002s
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:41.149209 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:41.136886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.1:44349)
0929 14:47:41.137516 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:41.137528 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:41.137567 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:47:41.137657 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:41.137667 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:41.137992 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:41.138233 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:41.139048 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.139921 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.141021 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.141463 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.141557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:41.144741 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:41.144753 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:41.144773 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:41.144844 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:41.147174 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.147669 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.147675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.147677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.147769 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.148074 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.148076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.148077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.148452 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:41.148663 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:41.148863 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 14:47:41.149026 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":167,"threads_started":1}
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:41.163017 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:41.153200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.98.62:38143)
0929 14:47:41.153376 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:41.153380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:41.153402 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:41.153613 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:41.153616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:41.153665 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:41.153747 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:41.154430 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.154981 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.156530 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:41.156761 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:41.156849 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:41.159596 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:41.159599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:41.159602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:41.159628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:41.161250 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.161754 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.161757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.161758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.161834 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:41.162078 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:41.162080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:41.162081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:41.162304 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:41.162414 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:41.162680 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 14:47:41.162795 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 14:47:41.163627 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970
I20260929 14:47:41.164652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4022
I20260929 14:47:41.172281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4156
I20260929 14:47:41.178586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4290
I20260929 14:47:41.185534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 3928
2026-09-29T14:47:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3243 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[4432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[4432](info): set up 2 sockets
Sep 29 14:47:41 dist-test-slave-5hd1 krb5kdc[4432](info): commencing operation
krb5kdc: starting...
W20260929 14:47:42.616722 392 mini_kdc.cc:121] Time spent starting KDC: real 1.406s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:42 dist-test-slave-5hd1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:42.755224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:42.882300 4445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:42.882645 4445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:42.882695 4445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:42.886986 4445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:42.887068 4445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:42.887087 4445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:42.887106 4445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:42.887122 4445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:42.892946 4445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:42.894367 4445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:42.895691 4445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:42.905714 4451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:42.905776 4445 server_base.cc:1034] running on GCE node
W20260929 14:47:42.905721 4450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:42.905731 4453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:42.906813 4445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:42.908262 4445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:42.909490 4445 hybrid_clock.cc:648] HybridClock initialized: now 1790693262909463 us; error 44 us; skew 500 ppm
Sep 29 14:47:42 dist-test-slave-5hd1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:42.912819 4445 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:42.913758 4445 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:42.913827 4445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:42.914110 4445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:42.916632 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "472ae5f4a8d44c0e842034fd28d9d01f"
format_stamp: "Formatted at 2026-09-29 14:47:42 on dist-test-slave-5hd1"
server_key: "747ad8d9c40e18636a9ec7faba2e922c"
server_key_iv: "a6d93a58a140d9e3cc2d30a01de72248"
server_key_version: "encryptionkey@0"
I20260929 14:47:42.917305 4445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "472ae5f4a8d44c0e842034fd28d9d01f"
format_stamp: "Formatted at 2026-09-29 14:47:42 on dist-test-slave-5hd1"
server_key: "747ad8d9c40e18636a9ec7faba2e922c"
server_key_iv: "a6d93a58a140d9e3cc2d30a01de72248"
server_key_version: "encryptionkey@0"
I20260929 14:47:42.921753 4445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:42.924674 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:42.926234 4445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:47:42.926393 4445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "472ae5f4a8d44c0e842034fd28d9d01f"
format_stamp: "Formatted at 2026-09-29 14:47:42 on dist-test-slave-5hd1"
server_key: "747ad8d9c40e18636a9ec7faba2e922c"
server_key_iv: "a6d93a58a140d9e3cc2d30a01de72248"
server_key_version: "encryptionkey@0"
I20260929 14:47:42.926520 4445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:42.940872 4445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:42.944571 4445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:42.944794 4445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:42.953545 4445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35601
I20260929 14:47:42.953869 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35601 every 8 connection(s)
I20260929 14:47:42.954397 4445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:42.957844 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:42.963016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4445
I20260929 14:47:42.963156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:42.963449 392 external_mini_cluster.cc:1428] Setting key 5e50f2f3ee24324940b4edd09004b806
I20260929 14:47:42.964393 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: Bootstrap starting.
I20260929 14:47:42.967010 4512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:42.968039 4512 log.cc:824] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:42.970403 4512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: No bootstrap required, opened a new log
Sep 29 14:47:42 dist-test-slave-5hd1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:42.974964 4512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } }
I20260929 14:47:42.975205 4512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:42.975268 4512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472ae5f4a8d44c0e842034fd28d9d01f, State: Initialized, Role: FOLLOWER
I20260929 14:47:42.975759 4512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } }
I20260929 14:47:42.975930 4512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:42.976011 4512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:42.976125 4512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:42.977082 4512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } }
I20260929 14:47:42.977490 4512 leader_election.cc:302] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472ae5f4a8d44c0e842034fd28d9d01f; no voters:
I20260929 14:47:42.977849 4512 leader_election.cc:288] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:42.978056 4517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:42.978435 4517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [term 1 LEADER]: Becoming Leader. State: Replica: 472ae5f4a8d44c0e842034fd28d9d01f, State: Running, Role: LEADER
I20260929 14:47:42.979080 4517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } }
I20260929 14:47:42.979259 4512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:42.981340 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:42.965221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.0.98.62:35601)
0929 14:47:42.965694 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:42.965724 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:42.965769 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:47:42.966575 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:42.966600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:42.967030 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:42.967451 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:42.968427 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:42.969512 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:42.970691 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:42.971224 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:42.971355 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:42.974543 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:42.974549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:42.974568 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:42.974602 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:42.977089 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:42.977672 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:42.977679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:42.977686 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:42.977860 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:42.978251 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:42.978254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:42.978256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:42.978944 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:42.979187 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:42.979649 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 14:47:42.980044 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":151,"threads_started":1}
I20260929 14:47:42.981567 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } } }
I20260929 14:47:42.982118 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:42.981829 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472ae5f4a8d44c0e842034fd28d9d01f. Latest consensus state: current_term: 1 leader_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472ae5f4a8d44c0e842034fd28d9d01f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35601 } } }
I20260929 14:47:42.982376 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f [sys.catalog]: This master's current role is: LEADER
W20260929 14:47:42.985061 4531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:42.985149 4531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:42.985234 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:42.985973 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:42.994154 4532 catalog_manager.cc:1347] Generated new cluster ID: f7243dd10cbd4488b7f677f59e59772e
I20260929 14:47:42.994270 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:43.014703 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:43.016379 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:43.033187 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: Generated new TSK 0
I20260929 14:47:43.034138 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:43.072234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:43.202391 4538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:43.202708 4538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:43.202759 4538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:43.207659 4538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:43.207743 4538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:43.207813 4538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:43.213661 4538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:43.215049 4538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:43.216367 4538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:43.227802 4544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.227845 4546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.227856 4543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:43.228427 4538 server_base.cc:1034] running on GCE node
I20260929 14:47:43.228999 4538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:43.229847 4538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:43.231079 4538 hybrid_clock.cc:648] HybridClock initialized: now 1790693263231058 us; error 60 us; skew 500 ppm
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:43.234736 4538 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:43.235693 4538 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:43.235805 4538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:43.236079 4538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:43.238783 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1a918b6474c4b5bb793c49f7d6bf41b"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "82f54f1ac69ffba14c9dc631b452c7c6"
server_key_iv: "18ab2c5c8bd0c8881dc77fc4c4381017"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.239482 4538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1a918b6474c4b5bb793c49f7d6bf41b"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "82f54f1ac69ffba14c9dc631b452c7c6"
server_key_iv: "18ab2c5c8bd0c8881dc77fc4c4381017"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.244015 4538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 14:47:43.247057 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.248925 4538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:43.249094 4538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "e1a918b6474c4b5bb793c49f7d6bf41b"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "82f54f1ac69ffba14c9dc631b452c7c6"
server_key_iv: "18ab2c5c8bd0c8881dc77fc4c4381017"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.249222 4538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:43.262533 4538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:43.266275 4538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:43.266510 4538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:43.267170 4538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:43.268110 4538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:43.268184 4538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.268301 4538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:43.268358 4538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.279413 4538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35613
I20260929 14:47:43.279434 4665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35613 every 8 connection(s)
I20260929 14:47:43.280231 4538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:43.280649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4538
I20260929 14:47:43.280825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:43.281216 392 external_mini_cluster.cc:1428] Setting key a8df6530ecb5d18b66b7ec1b9e78edec
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:43.303968 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.282555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41779 (local address 127.0.98.62:35601)
0929 14:47:43.282767 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.282773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.285023 (+ 2250us) server_negotiation.cc:407] Connection header received
0929 14:47:43.287578 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.287582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.287650 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.287758 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:43.289887 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.290453 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.292728 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.292989 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.293085 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.299460 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:43.299464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:43.299467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:43.299503 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:43.301236 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.302096 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.302102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.302107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.302201 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.302612 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.302616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.302617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.302847 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:43.302958 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.303594 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.303766 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 14:47:43.305102 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.283704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35601 (local address 127.0.98.1:41779)
0929 14:47:43.284762 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.284858 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.287257 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.288031 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.288044 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.288742 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:43.289683 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.289710 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.290626 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.290634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.292545 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.292558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.293200 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.293205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.293463 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.294366 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:43.294395 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:43.299086 (+ 4691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:43.301394 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.301405 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.301429 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.301944 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.302315 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.302325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.302328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.302501 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.302957 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:43.302968 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:43.303305 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.303537 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.303974 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":232,"threads_started":1}
I20260929 14:47:43.306804 4666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35601
I20260929 14:47:43.307206 4666 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:43.307857 4666 heartbeater.cc:511] Master 127.0.98.62:35601 requested a full tablet report, sending...
I20260929 14:47:43.309916 4471 ts_manager.cc:194] Registered new tserver with Master: e1a918b6474c4b5bb793c49f7d6bf41b (127.0.98.1:35613)
I20260929 14:47:43.311542 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:43.315872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:43.451634 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:43.452054 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:43.452149 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:43.456996 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:43.457129 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:43.457242 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:43.464627 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:43.466271 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:43.467808 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:43.480001 4679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.480037 4677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.480001 4676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:43.481297 4671 server_base.cc:1034] running on GCE node
I20260929 14:47:43.481935 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:43.482786 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:43.483990 4671 hybrid_clock.cc:648] HybridClock initialized: now 1790693263483966 us; error 46 us; skew 500 ppm
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:43.487278 4671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:43.488126 4671 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:43.488265 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:43.488523 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:43.490751 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "dbc79487bcbf4cd88c263dae80f6ac5e"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "dbebe4e5c929ed4acad7336dfaac028d"
server_key_iv: "7f56664c4e45a0688c202efb31109b43"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.491370 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dbc79487bcbf4cd88c263dae80f6ac5e"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "dbebe4e5c929ed4acad7336dfaac028d"
server_key_iv: "7f56664c4e45a0688c202efb31109b43"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.495954 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 14:47:43.498942 4685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.500662 4671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:43.500820 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "dbc79487bcbf4cd88c263dae80f6ac5e"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "dbebe4e5c929ed4acad7336dfaac028d"
server_key_iv: "7f56664c4e45a0688c202efb31109b43"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.500946 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:43.516117 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:43.520071 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:43.520335 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:43.520977 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:43.521868 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:43.521943 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.522018 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:43.522063 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.533270 4671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38387
I20260929 14:47:43.533370 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38387 every 8 connection(s)
I20260929 14:47:43.533998 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:43.534708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4671
I20260929 14:47:43.534857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:43.535271 392 external_mini_cluster.cc:1428] Setting key f1c1cecfe303c760e0fd1947d08628a7
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:43.556708 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.536625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56683 (local address 127.0.98.62:35601)
0929 14:47:43.536764 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.536767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.538270 (+ 1503us) server_negotiation.cc:407] Connection header received
0929 14:47:43.540338 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.540342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.540411 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.540518 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:43.542364 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.543140 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.545199 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.545455 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.545551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.551270 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:43.551279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:43.551287 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:43.551343 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:43.553540 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.554315 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.554319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.554321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.554429 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.554959 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.554962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.554963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.555356 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:43.555491 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.556345 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.556512 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:47:43.558071 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.537322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35601 (local address 127.0.98.2:56683)
0929 14:47:43.538051 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.538105 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.539918 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.540790 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.540804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.541367 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:43.542170 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.542195 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.543334 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.543341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.545028 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.545043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.545643 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.545650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.545916 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.546683 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:43.546710 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:43.550953 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:43.553732 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.553741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.553762 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.554167 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.554590 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.554596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.554601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.554805 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.555527 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:43.555537 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:43.555945 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.556254 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.556698 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":198,"threads_started":1}
I20260929 14:47:43.559792 4799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35601
I20260929 14:47:43.560308 4799 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:43.561064 4799 heartbeater.cc:511] Master 127.0.98.62:35601 requested a full tablet report, sending...
I20260929 14:47:43.562651 4471 ts_manager.cc:194] Registered new tserver with Master: dbc79487bcbf4cd88c263dae80f6ac5e (127.0.98.2:38387)
I20260929 14:47:43.563421 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:43.572887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:43.697386 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:43.697685 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:43.697731 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:43.701841 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:43.701907 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:43.701998 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:43.707497 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:43.709190 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:43.710629 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:43.721680 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.721680 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:43.721706 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:43.722072 4804 server_base.cc:1034] running on GCE node
I20260929 14:47:43.722694 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:43.723749 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:43.725004 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790693263724952 us; error 71 us; skew 500 ppm
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:43.728433 4804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:43.729328 4804 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:43.729398 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:43.729602 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:43.732095 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5bd2319090649289e8b319756840349"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "d90694a8c650cb66cac9c5abc8d38b54"
server_key_iv: "407b95548754e2311aa28fa6d044a93f"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.732806 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5bd2319090649289e8b319756840349"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "d90694a8c650cb66cac9c5abc8d38b54"
server_key_iv: "407b95548754e2311aa28fa6d044a93f"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.737959 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 14:47:43.741927 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.743821 4804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:43.744028 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "b5bd2319090649289e8b319756840349"
format_stamp: "Formatted at 2026-09-29 14:47:43 on dist-test-slave-5hd1"
server_key: "d90694a8c650cb66cac9c5abc8d38b54"
server_key_iv: "407b95548754e2311aa28fa6d044a93f"
server_key_version: "encryptionkey@0"
I20260929 14:47:43.744242 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:43.783806 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:43.788107 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:43.788441 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:43.789233 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:43.790309 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:43.790390 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.790465 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:43.790512 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:43.802645 4804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35611
I20260929 14:47:43.802668 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35611 every 8 connection(s)
I20260929 14:47:43.803589 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:43.811724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4804
I20260929 14:47:43.811879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:43.812237 392 external_mini_cluster.cc:1428] Setting key f32cbe82ec7ae14ce0e3ef81e2f9a17e
Sep 29 14:47:43 dist-test-slave-5hd1 krb5kdc[4432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:43.821408 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.806211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59881 (local address 127.0.98.62:35601)
0929 14:47:43.806387 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.806391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.807509 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 14:47:43.809014 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.809017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.809080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.809187 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:43.810493 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.811139 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.812854 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.813145 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.813274 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.817083 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:43.817086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:43.817089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:43.817116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:43.818860 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.819550 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.819553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.819554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.819636 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.820101 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.820104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.820105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.820379 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:43.820478 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.821144 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.821251 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:47:43.822607 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.806700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35601 (local address 127.0.98.3:59881)
0929 14:47:43.807309 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.807363 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.808778 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.809363 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.809373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.809755 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:43.810299 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.810312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.811278 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.811281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.812700 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.812710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.813570 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.813575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.813838 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.814800 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:43.814840 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:43.816864 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:43.819019 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.819026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.819045 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.819413 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.819766 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:43.819769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:43.819771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:43.819980 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:43.820515 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:43.820522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:43.820876 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.821071 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.821448 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":144,"threads_started":1}
I20260929 14:47:43.824417 4932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35601
I20260929 14:47:43.824759 4932 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:43.825403 4932 heartbeater.cc:511] Master 127.0.98.62:35601 requested a full tablet report, sending...
I20260929 14:47:43.827121 4471 ts_manager.cc:194] Registered new tserver with Master: b5bd2319090649289e8b319756840349 (127.0.98.3:35611)
I20260929 14:47:43.827879 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59881
I20260929 14:47:43.839654 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:47:43.850621 4515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.842091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.98.62:35601)
0929 14:47:43.842294 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.842299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.842541 (+ 242us) server_negotiation.cc:407] Connection header received
0929 14:47:43.842717 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.842721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.842780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.842871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:43.843711 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.844410 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.845254 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.845479 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.845575 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.846958 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:43.846961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:43.846964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:43.846987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:43.848753 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.849213 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.849216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.849218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.849311 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:43.849617 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:43.849620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:43.849621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:43.849950 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:43.850097 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.850288 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.850397 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:47:43.854452 4471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:43.857925 4471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:43.875675 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.868814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.0.98.2:38387)
0929 14:47:43.869276 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.869291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.869604 (+ 313us) server_negotiation.cc:407] Connection header received
0929 14:47:43.869872 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.869881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.870128 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.870286 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:43.870888 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872254 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.873786 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.874542 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.874673 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.874848 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.875289 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.875453 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":106,"threads_started":1}
I20260929 14:47:43.876384 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.868567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38387 (local address 127.0.0.1:46150)
0929 14:47:43.869438 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.869495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.869693 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.870521 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.870525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.870581 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:43.870770 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.870776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872411 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.872415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.873608 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.873619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.874880 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.874890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.875027 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.875078 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.875978 (+ 900us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.876163 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":90,"spinlock_wait_cycles":12288,"thread_start_us":155,"threads_started":1}
I20260929 14:47:43.876706 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.869060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.0.98.3:35611)
0929 14:47:43.869581 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.869596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.869624 (+ 28us) server_negotiation.cc:407] Connection header received
0929 14:47:43.869881 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.869887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.870096 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.870287 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:43.870960 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872249 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.874541 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.875370 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.875492 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.875633 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.876407 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.876522 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":172,"threads_started":1}
I20260929 14:47:43.876705 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.869676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35613 (local address 127.0.0.1:48190)
0929 14:47:43.870500 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.870513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.870615 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.871265 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.871268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.871288 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:43.871394 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.871401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872709 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.872712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.874244 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.874251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.876153 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.876158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.876297 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.876324 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.876420 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.876519 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":236,"spinlock_wait_cycles":384,"thread_start_us":101,"threads_started":1}
I20260929 14:47:43.877086 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.868431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35611 (local address 127.0.0.1:47564)
0929 14:47:43.869429 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.869496 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.869648 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.870515 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.870521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.870562 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:43.870800 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.870809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872442 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.872445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.874232 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.874245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.875717 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.875721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.875849 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.875895 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.875976 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.876995 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":81,"spinlock_wait_cycles":8832,"thread_start_us":97,"threads_started":1}
I20260929 14:47:43.877629 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.870005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.98.1:35613)
0929 14:47:43.870397 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.870405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.870631 (+ 226us) server_negotiation.cc:407] Connection header received
0929 14:47:43.870723 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.870728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.870930 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.871085 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:43.871535 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.872570 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.874933 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.875808 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.875921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.876080 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.876544 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.877504 (+ 960us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260929 14:47:43.880813 4733 tablet_service.cc:1467] Processing CreateTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc (DEFAULT_TABLE table=test-table [id=e097d599e4cc4759846fe4ab068eb7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:43.880813 4866 tablet_service.cc:1467] Processing CreateTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc (DEFAULT_TABLE table=test-table [id=e097d599e4cc4759846fe4ab068eb7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:43.881608 4600 tablet_service.cc:1467] Processing CreateTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc (DEFAULT_TABLE table=test-table [id=e097d599e4cc4759846fe4ab068eb7de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:43.882120 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63a41b520984eeabb8f9e7fd67abcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:43.882115 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63a41b520984eeabb8f9e7fd67abcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:43.882798 4600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63a41b520984eeabb8f9e7fd67abcdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:43.891036 4949 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Bootstrap starting.
I20260929 14:47:43.891315 4950 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Bootstrap starting.
I20260929 14:47:43.891958 4951 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: Bootstrap starting.
I20260929 14:47:43.894310 4949 tablet_bootstrap.cc:654] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:43.894624 4950 tablet_bootstrap.cc:654] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:43.894938 4951 tablet_bootstrap.cc:654] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:43.895865 4951 log.cc:824] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:43.895865 4949 log.cc:824] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:43.896140 4950 log.cc:824] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:43.898145 4951 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: No bootstrap required, opened a new log
I20260929 14:47:43.898398 4951 ts_tablet_manager.cc:1397] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:47:43.898432 4949 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: No bootstrap required, opened a new log
I20260929 14:47:43.898434 4950 tablet_bootstrap.cc:492] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: No bootstrap required, opened a new log
I20260929 14:47:43.898643 4949 ts_tablet_manager.cc:1397] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:47:43.898643 4950 ts_tablet_manager.cc:1397] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 14:47:43.902482 4951 raft_consensus.cc:348] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.902774 4951 raft_consensus.cc:374] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:43.902848 4951 raft_consensus.cc:729] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a918b6474c4b5bb793c49f7d6bf41b, State: Initialized, Role: FOLLOWER
I20260929 14:47:43.903136 4950 raft_consensus.cc:348] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.903136 4949 raft_consensus.cc:348] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.903491 4950 raft_consensus.cc:374] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:43.903503 4949 raft_consensus.cc:374] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:43.903452 4951 consensus_queue.cc:260] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.903579 4949 raft_consensus.cc:729] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc79487bcbf4cd88c263dae80f6ac5e, State: Initialized, Role: FOLLOWER
I20260929 14:47:43.903641 4950 raft_consensus.cc:729] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bd2319090649289e8b319756840349, State: Initialized, Role: FOLLOWER
I20260929 14:47:43.904268 4666 heartbeater.cc:503] Master 127.0.98.62:35601 was elected leader, sending a full tablet report...
I20260929 14:47:43.904202 4949 consensus_queue.cc:260] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.904201 4950 consensus_queue.cc:260] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.904717 4951 ts_tablet_manager.cc:1428] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:47:43.905359 4950 ts_tablet_manager.cc:1428] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:47:43.905359 4949 ts_tablet_manager.cc:1428] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:47:43.905624 4799 heartbeater.cc:503] Master 127.0.98.62:35601 was elected leader, sending a full tablet report...
I20260929 14:47:43.905835 4932 heartbeater.cc:503] Master 127.0.98.62:35601 was elected leader, sending a full tablet report...
I20260929 14:47:43.933535 4956 raft_consensus.cc:482] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:43.933843 4956 raft_consensus.cc:504] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.935324 4956 leader_election.cc:288] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc79487bcbf4cd88c263dae80f6ac5e (127.0.98.2:38387), e1a918b6474c4b5bb793c49f7d6bf41b (127.0.98.1:35613)
I20260929 14:47:43.939965 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.935742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35613 (local address 127.0.98.3:54659)
0929 14:47:43.935961 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.935980 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.936107 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.936565 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.936572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.936599 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:43.936750 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.936757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.937734 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.937738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.938648 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.938657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.939461 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.939465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.939578 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.939607 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.939714 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.939797 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 14:47:43.940101 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.935818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54659 (local address 127.0.98.1:35613)
0929 14:47:43.935972 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.935978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.936049 (+ 71us) server_negotiation.cc:407] Connection header received
0929 14:47:43.936301 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.936307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.936377 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.936503 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:43.936936 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.937617 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.938782 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.939326 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.939413 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.939500 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.939786 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.939910 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:47:43.940490 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.935843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56615 (local address 127.0.98.2:38387)
0929 14:47:43.936045 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.936050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.936593 (+ 543us) server_negotiation.cc:407] Connection header received
0929 14:47:43.936707 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.936712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.936792 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.936885 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:43.937250 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.937833 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.939097 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.939651 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.939775 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.939837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.940107 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.940356 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:47:43.941048 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63a41b520984eeabb8f9e7fd67abcdc" candidate_uuid: "b5bd2319090649289e8b319756840349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" is_pre_election: true
I20260929 14:47:43.941147 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.935756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38387 (local address 127.0.98.3:56615)
0929 14:47:43.936456 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:43.936476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:43.936592 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:43.936936 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:43.936939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:43.936960 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:43.937117 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.937123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.937976 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.938007 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.938938 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:43.938948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.939800 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.939804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:43.939919 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.939945 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:47:43.940907 (+ 962us) client_negotiation.cc:240] Negotiation successful
0929 14:47:43.941010 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":171,"thread_start_us":164,"threads_started":1}
I20260929 14:47:43.941565 4620 raft_consensus.cc:2454] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5bd2319090649289e8b319756840349 in term 0.
I20260929 14:47:43.941936 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63a41b520984eeabb8f9e7fd67abcdc" candidate_uuid: "b5bd2319090649289e8b319756840349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" is_pre_election: true
I20260929 14:47:43.942236 4819 leader_election.cc:302] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5bd2319090649289e8b319756840349, e1a918b6474c4b5bb793c49f7d6bf41b; no voters:
I20260929 14:47:43.942334 4752 raft_consensus.cc:2454] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5bd2319090649289e8b319756840349 in term 0.
I20260929 14:47:43.942572 4956 raft_consensus.cc:2781] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:43.942695 4956 raft_consensus.cc:482] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:43.942760 4956 raft_consensus.cc:3037] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:43.944042 4956 raft_consensus.cc:504] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.944527 4956 leader_election.cc:288] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [CANDIDATE]: Term 1 election: Requested vote from peers dbc79487bcbf4cd88c263dae80f6ac5e (127.0.98.2:38387), e1a918b6474c4b5bb793c49f7d6bf41b (127.0.98.1:35613)
I20260929 14:47:43.944995 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63a41b520984eeabb8f9e7fd67abcdc" candidate_uuid: "b5bd2319090649289e8b319756840349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e"
I20260929 14:47:43.945006 4620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63a41b520984eeabb8f9e7fd67abcdc" candidate_uuid: "b5bd2319090649289e8b319756840349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b"
I20260929 14:47:43.945129 4752 raft_consensus.cc:3037] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:43.945183 4620 raft_consensus.cc:3037] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:43.946416 4752 raft_consensus.cc:2454] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5bd2319090649289e8b319756840349 in term 1.
I20260929 14:47:43.946419 4620 raft_consensus.cc:2454] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5bd2319090649289e8b319756840349 in term 1.
I20260929 14:47:43.946779 4820 leader_election.cc:302] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5bd2319090649289e8b319756840349, dbc79487bcbf4cd88c263dae80f6ac5e; no voters:
I20260929 14:47:43.947005 4956 raft_consensus.cc:2781] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:43.947410 4956 raft_consensus.cc:686] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 1 LEADER]: Becoming Leader. State: Replica: b5bd2319090649289e8b319756840349, State: Running, Role: LEADER
I20260929 14:47:43.947861 4956 consensus_queue.cc:237] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } }
I20260929 14:47:43.952718 4474 catalog_manager.cc:5495] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5bd2319090649289e8b319756840349 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b5bd2319090649289e8b319756840349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5bd2319090649289e8b319756840349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:43.965853 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:43.961669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.3:35611)
0929 14:47:43.961789 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:43.961792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:43.961872 (+ 80us) server_negotiation.cc:407] Connection header received
0929 14:47:43.962036 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:43.962038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:43.962091 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:43.962187 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:43.962497 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.963005 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.964440 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:43.964649 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:43.964731 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:43.965054 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:43.965460 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:43.965555 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:43.965649 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 14:47:43.965692 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:47:43.973243 4752 raft_consensus.cc:1261] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 1 FOLLOWER]: Refusing update from remote peer b5bd2319090649289e8b319756840349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:43.973228 4620 raft_consensus.cc:1261] T f63a41b520984eeabb8f9e7fd67abcdc P e1a918b6474c4b5bb793c49f7d6bf41b [term 1 FOLLOWER]: Refusing update from remote peer b5bd2319090649289e8b319756840349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:43.974143 4959 consensus_queue.cc:1035] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc79487bcbf4cd88c263dae80f6ac5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:43.974426 4956 consensus_queue.cc:1035] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a918b6474c4b5bb793c49f7d6bf41b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:43.981868 4962 mvcc.cc:205] Tried to move back new op lower bound from 7334679609224630272 to 7334679609134252032. Current Snapshot: MvccSnapshot[applied={T|T < 7334679609224630272}]
I20260929 14:47:43.995460 4474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:43.995725 4474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:43.999428 4474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 472ae5f4a8d44c0e842034fd28d9d01f: Sending DeleteTablet for 3 replicas of tablet f63a41b520984eeabb8f9e7fd67abcdc
I20260929 14:47:44.000910 4866 tablet_service.cc:1514] Processing DeleteTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47564
I20260929 14:47:44.000993 4733 tablet_service.cc:1514] Processing DeleteTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46150
I20260929 14:47:44.001693 4600 tablet_service.cc:1514] Processing DeleteTablet for tablet f63a41b520984eeabb8f9e7fd67abcdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48190
I20260929 14:47:44.001749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4538
I20260929 14:47:44.002032 4971 tablet_replica.cc:330] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: stopping tablet replica
I20260929 14:47:44.002524 4971 raft_consensus.cc:2229] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:44.003304 4971 raft_consensus.cc:2258] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:44.004134 4972 tablet_replica.cc:330] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: stopping tablet replica
I20260929 14:47:44.004660 4972 raft_consensus.cc:2229] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:44.005505 4972 raft_consensus.cc:2258] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:44.006256 4971 ts_tablet_manager.cc:1905] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:44.007400 4972 ts_tablet_manager.cc:1905] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:44.010982 4971 ts_tablet_manager.cc:1918] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:44.011324 4971 log.cc:1196] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/f63a41b520984eeabb8f9e7fd67abcdc
I20260929 14:47:44.011816 4971 ts_tablet_manager.cc:1939] T f63a41b520984eeabb8f9e7fd67abcdc P dbc79487bcbf4cd88c263dae80f6ac5e: Deleting consensus metadata
W20260929 14:47:44.012403 4460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35613 (error 108)
I20260929 14:47:44.012251 4972 ts_tablet_manager.cc:1918] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:44.012964 4972 log.cc:1196] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/f63a41b520984eeabb8f9e7fd67abcdc
I20260929 14:47:44.013432 4972 ts_tablet_manager.cc:1939] T f63a41b520984eeabb8f9e7fd67abcdc P b5bd2319090649289e8b319756840349: Deleting consensus metadata
I20260929 14:47:44.013496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4671
I20260929 14:47:44.016220 4460 catalog_manager.cc:4841] TS b5bd2319090649289e8b319756840349 (127.0.98.3:35611): tablet f63a41b520984eeabb8f9e7fd67abcdc (table test-table [id=e097d599e4cc4759846fe4ab068eb7de]) successfully deleted
I20260929 14:47:44.016983 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.016658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35613 (local address 127.0.0.1:48200)
0929 14:47:44.016813 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.016894 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 14:47:44.017263 4460 catalog_manager.cc:4584] TS e1a918b6474c4b5bb793c49f7d6bf41b (127.0.98.1:35613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f63a41b520984eeabb8f9e7fd67abcdc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35613: connect: Connection refused (error 111)
I20260929 14:47:44.021107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4804
I20260929 14:47:44.022939 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.022446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38387 (local address 127.0.0.1:46164)
0929 14:47:44.022716 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.022797 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 14:47:44.029405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4445
2026-09-29T14:47:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2843 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T14:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:44Z Disabled control of system clock
I20260929 14:47:44.063457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46739
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:47:44.176002 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:44.176393 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:44.176441 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:44.180390 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:44.180462 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:44.180477 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:44.180495 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:44.180510 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:44.185329 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:44.186451 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:44.187830 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:44.197222 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.197310 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.197232 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:44.197877 4977 server_base.cc:1034] running on GCE node
I20260929 14:47:44.198518 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:44.199730 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:44.200946 4977 hybrid_clock.cc:648] HybridClock initialized: now 1790693264200919 us; error 51 us; skew 500 ppm
I20260929 14:47:44.203001 4977 webserver.cc:460] Webserver started at http://127.0.98.62:41033/ using document root <none> and password file <none>
I20260929 14:47:44.203630 4977 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:44.203742 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:44.203989 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:44.206189 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "b9c8c7c8c1cf757a1259b8a12aee1169"
server_key_iv: "5eb37c995170a32f89c61a49b6d84d6b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.206774 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "b9c8c7c8c1cf757a1259b8a12aee1169"
server_key_iv: "5eb37c995170a32f89c61a49b6d84d6b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.210896 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 14:47:44.213722 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.215265 4977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:44.215423 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "b9c8c7c8c1cf757a1259b8a12aee1169"
server_key_iv: "5eb37c995170a32f89c61a49b6d84d6b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.215544 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:44.220419 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:44.221066 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:44.221257 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:44.228698 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46739
I20260929 14:47:44.228770 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46739 every 8 connection(s)
I20260929 14:47:44.229739 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:44.230513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 4977
I20260929 14:47:44.230652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:44.231005 392 external_mini_cluster.cc:1428] Setting key 93e2ede2ebe55f503873928b00c43b43
I20260929 14:47:44.233819 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:44.237684 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.232753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.62:46739)
0929 14:47:44.233964 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.233987 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.234033 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:47:44.234720 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.234738 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.235135 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.235496 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.236378 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.236384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.236396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.236406 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.236462 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.236479 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.236630 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.236700 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.236953 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":584,"threads_started":1}
I20260929 14:47:44.241688 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: Bootstrap starting.
I20260929 14:47:44.244627 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:44.245647 5044 log.cc:824] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:44.248374 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: No bootstrap required, opened a new log
I20260929 14:47:44.252543 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260929 14:47:44.252918 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:44.252982 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8baa7ef0dcd4103a6a9a89373e83c5b, State: Initialized, Role: FOLLOWER
I20260929 14:47:44.253610 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260929 14:47:44.253790 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:44.253865 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:44.253975 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:44.255119 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260929 14:47:44.255544 5044 leader_election.cc:302] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8baa7ef0dcd4103a6a9a89373e83c5b; no voters:
I20260929 14:47:44.255940 5044 leader_election.cc:288] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:44.256242 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:44.256577 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [term 1 LEADER]: Becoming Leader. State: Replica: d8baa7ef0dcd4103a6a9a89373e83c5b, State: Running, Role: LEADER
I20260929 14:47:44.257004 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } }
I20260929 14:47:44.257532 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:44.259089 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } } }
I20260929 14:47:44.259238 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:44.259542 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8baa7ef0dcd4103a6a9a89373e83c5b. Latest consensus state: current_term: 1 leader_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa7ef0dcd4103a6a9a89373e83c5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46739 } } }
I20260929 14:47:44.259696 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:44.259931 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:44.262956 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:44.269054 5058 catalog_manager.cc:1347] Generated new cluster ID: f89e3d9c1b1245ae894a1f1ee23b0441
I20260929 14:47:44.269131 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:44.290108 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:44.290625 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:44.299975 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: Generated new TSK 0
I20260929 14:47:44.301136 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:47:44.310951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--builtin_ntp_servers=127.0.98.20:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:47:44.425372 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:44.425657 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:44.425700 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:44.429999 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:44.430063 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:44.430124 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:44.434875 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:44.435890 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:44.437305 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:44.448212 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.448205 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.448153 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:44.448269 5068 server_base.cc:1034] running on GCE node
I20260929 14:47:44.448921 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:44.449682 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:44.450906 5068 hybrid_clock.cc:648] HybridClock initialized: now 1790693264450880 us; error 69 us; skew 500 ppm
I20260929 14:47:44.453209 5068 webserver.cc:460] Webserver started at http://127.0.98.1:33245/ using document root <none> and password file <none>
I20260929 14:47:44.453910 5068 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:44.454007 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:44.454288 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:44.456790 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ea453b89fbe412cba0fb0acc9b45bc9"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "0afdbc30f287207920df9317119a23ba"
server_key_iv: "b17141e5fdf588261b313fc1bb2b26d1"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.457469 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea453b89fbe412cba0fb0acc9b45bc9"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "0afdbc30f287207920df9317119a23ba"
server_key_iv: "b17141e5fdf588261b313fc1bb2b26d1"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.461963 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 14:47:44.464756 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.466228 5068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:44.466374 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "0ea453b89fbe412cba0fb0acc9b45bc9"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "0afdbc30f287207920df9317119a23ba"
server_key_iv: "b17141e5fdf588261b313fc1bb2b26d1"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.466503 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:44.471834 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:44.472810 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:44.473016 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:44.473762 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:44.474808 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:44.474860 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.474903 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:44.474918 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.485867 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42845
I20260929 14:47:44.485896 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42845 every 8 connection(s)
I20260929 14:47:44.486990 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:44.493748 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.489112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48719 (local address 127.0.98.62:46739)
0929 14:47:44.489310 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.489316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.490479 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 14:47:44.491909 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.491912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.491947 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.492030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.493335 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.493338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.493340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.493350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.493369 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.493377 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.493452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.493607 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.493642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:47:44.494823 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.489506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.1:48719)
0929 14:47:44.490298 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.490345 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.491695 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.492274 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.492282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.492381 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.493034 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.493036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.493085 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.493163 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.493460 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.493466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.493471 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.493639 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.493911 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260929 14:47:44.496129 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260929 14:47:44.496393 5196 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:44.496805 5196 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260929 14:47:44.497309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5068
I20260929 14:47:44.497450 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:44.497777 392 external_mini_cluster.cc:1428] Setting key 20d7961ad8ad0a530af5b93d3bb00990
I20260929 14:47:44.498517 5008 ts_manager.cc:194] Registered new tserver with Master: 0ea453b89fbe412cba0fb0acc9b45bc9 (127.0.98.1:42845)
I20260929 14:47:44.500094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--builtin_ntp_servers=127.0.98.20:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:47:44.614018 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:44.614269 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:44.614307 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:44.618295 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:44.618352 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:44.618412 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:44.623202 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:44.624280 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:44.625609 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:44.635571 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.635572 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:44.635622 5199 server_base.cc:1034] running on GCE node
W20260929 14:47:44.635589 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:44.636402 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:44.637076 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:44.638270 5199 hybrid_clock.cc:648] HybridClock initialized: now 1790693264638247 us; error 40 us; skew 500 ppm
I20260929 14:47:44.640214 5199 webserver.cc:460] Webserver started at http://127.0.98.2:38609/ using document root <none> and password file <none>
I20260929 14:47:44.640844 5199 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:44.640908 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:44.641119 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:44.643137 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "525da0e1e4c34c4397ad550e0c1374ab"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "7ba7263e5d6071b918eb5a872056e785"
server_key_iv: "cec2f56adaee08fb48eedb0207d16805"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.643749 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "525da0e1e4c34c4397ad550e0c1374ab"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "7ba7263e5d6071b918eb5a872056e785"
server_key_iv: "cec2f56adaee08fb48eedb0207d16805"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.647763 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:47:44.650528 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.652029 5199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:44.652287 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "525da0e1e4c34c4397ad550e0c1374ab"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "7ba7263e5d6071b918eb5a872056e785"
server_key_iv: "cec2f56adaee08fb48eedb0207d16805"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.652417 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:44.657172 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:44.657821 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:44.658023 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:44.658674 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:44.659657 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:44.659706 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.659745 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:44.659759 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.669858 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36827
I20260929 14:47:44.669958 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36827 every 8 connection(s)
I20260929 14:47:44.670902 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:44.676707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5199
I20260929 14:47:44.676811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:44.677098 392 external_mini_cluster.cc:1428] Setting key 518d0c14774a5b9332c170ad0a7ccdaf
I20260929 14:47:44.678267 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.673284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38765 (local address 127.0.98.62:46739)
0929 14:47:44.673489 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.673494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.674641 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 14:47:44.676097 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.676099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.676139 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.676289 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.677739 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.677742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.677745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.677764 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.677806 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.677817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.677904 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.678106 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.678147 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:47:44.679279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--builtin_ntp_servers=127.0.98.20:40539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 14:47:44.679584 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.673655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.2:38765)
0929 14:47:44.674457 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.674508 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.675873 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.676470 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.676478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.676583 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.677409 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.677411 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.677469 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.677583 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.677930 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.677939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.677944 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.678140 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.678447 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260929 14:47:44.681100 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260929 14:47:44.681363 5327 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:44.681783 5327 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260929 14:47:44.683028 5008 ts_manager.cc:194] Registered new tserver with Master: 525da0e1e4c34c4397ad550e0c1374ab (127.0.98.2:36827)
W20260929 14:47:44.796280 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:44.796551 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:44.796589 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:44.800730 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:44.800786 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:44.800839 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:44.805439 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:44.806417 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:44.807711 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:44.817734 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.817777 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:44.817734 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:44.818008 5330 server_base.cc:1034] running on GCE node
I20260929 14:47:44.818605 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:44.819489 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:44.820667 5330 hybrid_clock.cc:648] HybridClock initialized: now 1790693264820632 us; error 69 us; skew 500 ppm
I20260929 14:47:44.822894 5330 webserver.cc:460] Webserver started at http://127.0.98.3:33549/ using document root <none> and password file <none>
I20260929 14:47:44.823585 5330 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:44.823652 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:44.823892 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:44.826257 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3e99fa998d04b64b4c0509bb1f17ff5"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "6cf3962879fb237714b355d50098b65b"
server_key_iv: "f8be68ab09180961f5823c2f4fa8a36b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.826943 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3e99fa998d04b64b4c0509bb1f17ff5"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "6cf3962879fb237714b355d50098b65b"
server_key_iv: "f8be68ab09180961f5823c2f4fa8a36b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.831282 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:47:44.834409 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.835944 5330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:44.836099 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "a3e99fa998d04b64b4c0509bb1f17ff5"
format_stamp: "Formatted at 2026-09-29 14:47:44 on dist-test-slave-5hd1"
server_key: "6cf3962879fb237714b355d50098b65b"
server_key_iv: "f8be68ab09180961f5823c2f4fa8a36b"
server_key_version: "encryptionkey@0"
I20260929 14:47:44.836294 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:44.841373 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:44.842065 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:44.842270 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:44.842908 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:44.843964 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:44.844014 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.844075 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:44.844133 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:44.855238 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36675
I20260929 14:47:44.855335 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36675 every 8 connection(s)
I20260929 14:47:44.856457 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:44.857268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5330
I20260929 14:47:44.857398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:44.857686 392 external_mini_cluster.cc:1428] Setting key 46d9bc0253d1095d3e997fff2ab29c71
I20260929 14:47:44.863683 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.858728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37819 (local address 127.0.98.62:46739)
0929 14:47:44.858900 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.858906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.860683 (+ 1777us) server_negotiation.cc:407] Connection header received
0929 14:47:44.861991 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.861993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.862037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.862113 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.863237 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.863241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.863243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.863253 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.863276 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.863284 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.863365 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.863518 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.863562 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 14:47:44.864643 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.859310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46739 (local address 127.0.98.3:37819)
0929 14:47:44.860253 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.860578 (+ 325us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.861799 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.862249 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.862256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.862345 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.862964 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.862966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.863021 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.863089 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.863370 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.863376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.863380 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.863539 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.863735 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":180,"threads_started":1}
I20260929 14:47:44.865896 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46739
I20260929 14:47:44.866150 5458 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:44.866597 5458 heartbeater.cc:511] Master 127.0.98.62:46739 requested a full tablet report, sending...
I20260929 14:47:44.867728 5008 ts_manager.cc:194] Registered new tserver with Master: a3e99fa998d04b64b4c0509bb1f17ff5 (127.0.98.3:36675)
I20260929 14:47:44.872426 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:47:44.876606 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.874706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.62:46739)
0929 14:47:44.874887 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.874891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.875113 (+ 222us) server_negotiation.cc:407] Connection header received
0929 14:47:44.875193 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.875195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.875231 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.875312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.876094 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.876097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.876100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.876115 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.876141 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.876151 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.876290 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.876429 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.876490 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:47:44.879657 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:44.883419 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:44.897109 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.893218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36827 (local address 127.0.0.1:33658)
0929 14:47:44.894257 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.894311 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.894424 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.895472 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.895478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.895558 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.896208 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.896211 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.896235 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.896319 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.896741 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.896749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.896755 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.896868 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.896942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":86,"threads_started":1}
I20260929 14:47:44.897562 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.893225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36675 (local address 127.0.0.1:40458)
0929 14:47:44.894711 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.894728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.894869 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.896064 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.896069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.896089 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.896850 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.896852 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.896863 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.896890 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.897288 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.897293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.897297 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.897407 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.897440 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":143,"threads_started":1}
I20260929 14:47:44.897773 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.894047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.0.98.3:36675)
0929 14:47:44.894557 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.894575 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.894806 (+ 231us) server_negotiation.cc:407] Connection header received
0929 14:47:44.895023 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.895029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.895199 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.895294 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.897029 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.897034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.897044 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.897055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.897094 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.897123 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.897210 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.897454 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.897561 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":182,"threads_started":1}
I20260929 14:47:44.897773 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.894024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.0.98.1:42845)
0929 14:47:44.894685 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.894695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.894948 (+ 253us) server_negotiation.cc:407] Connection header received
0929 14:47:44.895015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.895020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.895258 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.895438 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.897087 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.897094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.897110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.897124 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.897177 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.897211 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.897302 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.897489 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.897560 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":120,"threads_started":1}
I20260929 14:47:44.898226 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.894536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.0.98.2:36827)
0929 14:47:44.894905 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:44.894921 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:44.894943 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:44.895013 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:44.895028 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:44.895257 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:44.895438 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:44.896492 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:44.896500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:44.896513 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:44.896530 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:44.896588 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:44.896640 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:44.896767 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:44.897835 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0929 14:47:44.898036 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":127,"threads_started":1}
I20260929 14:47:44.898415 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:44.893530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42845 (local address 127.0.0.1:54032)
0929 14:47:44.894814 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:44.894828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:44.894910 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:44.896233 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:44.896237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:44.896290 (+ 53us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:44.896893 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:44.896895 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:44.896908 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:44.896939 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:44.897375 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:44.897378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:44.897381 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:47:44.898097 (+ 716us) client_negotiation.cc:240] Negotiation successful
0929 14:47:44.898138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":137,"threads_started":1}
I20260929 14:47:44.901381 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 7428351e1433462195d7432c7aa385d3 (DEFAULT_TABLE table=test-table [id=9a020542129e45f5874bae6ddc7a996e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:44.901394 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 7428351e1433462195d7432c7aa385d3 (DEFAULT_TABLE table=test-table [id=9a020542129e45f5874bae6ddc7a996e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:44.901396 5130 tablet_service.cc:1467] Processing CreateTablet for tablet 7428351e1433462195d7432c7aa385d3 (DEFAULT_TABLE table=test-table [id=9a020542129e45f5874bae6ddc7a996e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:44.902537 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7428351e1433462195d7432c7aa385d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:44.902563 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7428351e1433462195d7432c7aa385d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:44.902563 5130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7428351e1433462195d7432c7aa385d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:44.909775 5475 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Bootstrap starting.
I20260929 14:47:44.911187 5477 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Bootstrap starting.
I20260929 14:47:44.911187 5476 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Bootstrap starting.
I20260929 14:47:44.912391 5475 tablet_bootstrap.cc:654] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:44.913228 5475 log.cc:824] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:44.913547 5476 tablet_bootstrap.cc:654] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:44.913547 5477 tablet_bootstrap.cc:654] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:44.914475 5477 log.cc:824] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:44.914475 5476 log.cc:824] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:44.915549 5475 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: No bootstrap required, opened a new log
I20260929 14:47:44.915768 5475 ts_tablet_manager.cc:1397] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 14:47:44.916443 5477 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: No bootstrap required, opened a new log
I20260929 14:47:44.916507 5476 tablet_bootstrap.cc:492] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: No bootstrap required, opened a new log
I20260929 14:47:44.916648 5477 ts_tablet_manager.cc:1397] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 14:47:44.916693 5476 ts_tablet_manager.cc:1397] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 14:47:44.920264 5477 raft_consensus.cc:348] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.920534 5477 raft_consensus.cc:374] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:44.920588 5477 raft_consensus.cc:729] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea453b89fbe412cba0fb0acc9b45bc9, State: Initialized, Role: FOLLOWER
I20260929 14:47:44.920895 5475 raft_consensus.cc:348] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.921056 5477 consensus_queue.cc:260] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.921187 5475 raft_consensus.cc:374] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:44.921249 5475 raft_consensus.cc:729] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e99fa998d04b64b4c0509bb1f17ff5, State: Initialized, Role: FOLLOWER
I20260929 14:47:44.921639 5476 raft_consensus.cc:348] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.921766 5475 consensus_queue.cc:260] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.921875 5476 raft_consensus.cc:374] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:44.921918 5476 raft_consensus.cc:729] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525da0e1e4c34c4397ad550e0c1374ab, State: Initialized, Role: FOLLOWER
I20260929 14:47:44.922058 5477 ts_tablet_manager.cc:1428] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 14:47:44.922330 5476 consensus_queue.cc:260] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:44.922490 5196 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
I20260929 14:47:44.922833 5475 ts_tablet_manager.cc:1428] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:47:44.922960 5458 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
I20260929 14:47:44.923211 5476 ts_tablet_manager.cc:1428] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:44.923660 5327 heartbeater.cc:503] Master 127.0.98.62:46739 was elected leader, sending a full tablet report...
W20260929 14:47:44.988396 5197 tablet.cc:2369] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:47:45.107811 5459 tablet.cc:2369] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:45.165971 5481 raft_consensus.cc:482] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:45.166219 5481 raft_consensus.cc:504] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:45.167481 5481 leader_election.cc:288] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e99fa998d04b64b4c0509bb1f17ff5 (127.0.98.3:36675), 525da0e1e4c34c4397ad550e0c1374ab (127.0.98.2:36827)
I20260929 14:47:45.170027 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.167906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50867 (local address 127.0.98.2:36827)
0929 14:47:45.168062 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.168066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.168364 (+ 298us) server_negotiation.cc:407] Connection header received
0929 14:47:45.168487 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.168491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.168560 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.168654 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:45.169505 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:45.169509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:45.169518 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:45.169531 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:45.169554 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:45.169562 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:45.169677 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:45.169815 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 14:47:45.169864 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 14:47:45.170061 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.168000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45981 (local address 127.0.98.3:36675)
0929 14:47:45.168236 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.168242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.168521 (+ 279us) server_negotiation.cc:407] Connection header received
0929 14:47:45.168657 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.168660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.168701 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.168799 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:45.169662 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:45.169665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:45.169668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:45.169678 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:45.169698 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:45.169708 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:45.169831 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:45.169911 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 14:47:45.169947 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 14:47:45.170037 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.167864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36827 (local address 127.0.98.1:50867)
0929 14:47:45.168234 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:45.168250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:45.168379 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:45.168698 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:45.168702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:45.168722 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:45.169315 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:45.169317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:45.169328 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:45.169355 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:45.169681 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:45.169685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:45.169688 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:47:45.169768 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:47:45.169833 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
I20260929 14:47:45.170063 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.167927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36675 (local address 127.0.98.1:45981)
0929 14:47:45.168393 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:45.168410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:45.168531 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:45.168832 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:45.168834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:45.168848 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:45.169503 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:47:45.169504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:45.169514 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:47:45.169540 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:45.169828 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:45.169830 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:45.169833 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:47:45.169911 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:47:45.169937 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":146,"threads_started":1}
I20260929 14:47:45.170794 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7428351e1433462195d7432c7aa385d3" candidate_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" is_pre_election: true
I20260929 14:47:45.170950 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7428351e1433462195d7432c7aa385d3" candidate_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525da0e1e4c34c4397ad550e0c1374ab" is_pre_election: true
I20260929 14:47:45.171103 5412 raft_consensus.cc:2454] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea453b89fbe412cba0fb0acc9b45bc9 in term 0.
I20260929 14:47:45.171386 5281 raft_consensus.cc:2454] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea453b89fbe412cba0fb0acc9b45bc9 in term 0.
I20260929 14:47:45.171656 5084 leader_election.cc:302] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea453b89fbe412cba0fb0acc9b45bc9, a3e99fa998d04b64b4c0509bb1f17ff5; no voters:
I20260929 14:47:45.172014 5481 raft_consensus.cc:2781] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:45.172115 5481 raft_consensus.cc:482] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:45.172161 5481 raft_consensus.cc:3037] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:47:45.172290 5328 tablet.cc:2369] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:45.173614 5481 raft_consensus.cc:504] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:45.174062 5481 leader_election.cc:288] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [CANDIDATE]: Term 1 election: Requested vote from peers a3e99fa998d04b64b4c0509bb1f17ff5 (127.0.98.3:36675), 525da0e1e4c34c4397ad550e0c1374ab (127.0.98.2:36827)
I20260929 14:47:45.174392 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7428351e1433462195d7432c7aa385d3" candidate_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5"
I20260929 14:47:45.174453 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7428351e1433462195d7432c7aa385d3" candidate_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525da0e1e4c34c4397ad550e0c1374ab"
I20260929 14:47:45.174543 5412 raft_consensus.cc:3037] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:45.174611 5281 raft_consensus.cc:3037] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:45.175745 5412 raft_consensus.cc:2454] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea453b89fbe412cba0fb0acc9b45bc9 in term 1.
I20260929 14:47:45.175751 5281 raft_consensus.cc:2454] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea453b89fbe412cba0fb0acc9b45bc9 in term 1.
I20260929 14:47:45.176040 5084 leader_election.cc:302] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea453b89fbe412cba0fb0acc9b45bc9, a3e99fa998d04b64b4c0509bb1f17ff5; no voters:
I20260929 14:47:45.176281 5481 raft_consensus.cc:2781] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:45.176548 5481 raft_consensus.cc:686] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 0ea453b89fbe412cba0fb0acc9b45bc9, State: Running, Role: LEADER
I20260929 14:47:45.176929 5481 consensus_queue.cc:237] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } }
I20260929 14:47:45.180882 5007 catalog_manager.cc:5495] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ea453b89fbe412cba0fb0acc9b45bc9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea453b89fbe412cba0fb0acc9b45bc9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42845 } health_report { overall_health: HEALTHY } } }
I20260929 14:47:45.191229 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.189405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.0.98.1:42845)
0929 14:47:45.189552 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.189558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.189662 (+ 104us) server_negotiation.cc:407] Connection header received
0929 14:47:45.189875 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.189877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.189943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.190068 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:45.190803 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:45.190806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:45.190809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:45.190827 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:45.190844 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:45.190852 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:45.190950 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:45.191019 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 14:47:45.191070 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:47:45.198036 5412 raft_consensus.cc:1261] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 1 FOLLOWER]: Refusing update from remote peer 0ea453b89fbe412cba0fb0acc9b45bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:45.198141 5281 raft_consensus.cc:1261] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 1 FOLLOWER]: Refusing update from remote peer 0ea453b89fbe412cba0fb0acc9b45bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:45.198969 5486 consensus_queue.cc:1035] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e99fa998d04b64b4c0509bb1f17ff5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:45.199240 5481 consensus_queue.cc:1035] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525da0e1e4c34c4397ad550e0c1374ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:45.210047 5490 mvcc.cc:205] Tried to move back new op lower bound from 7334679614242729984 to 7334679614167855104. Current Snapshot: MvccSnapshot[applied={T|T < 7334679614242729984}]
I20260929 14:47:45.217602 5007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:45.217862 5007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:45.220840 5007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8baa7ef0dcd4103a6a9a89373e83c5b: Sending DeleteTablet for 3 replicas of tablet 7428351e1433462195d7432c7aa385d3
I20260929 14:47:45.221832 5130 tablet_service.cc:1514] Processing DeleteTablet for tablet 7428351e1433462195d7432c7aa385d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:45 UTC) from {username='slave'} at 127.0.0.1:54032
I20260929 14:47:45.221843 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 7428351e1433462195d7432c7aa385d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:45 UTC) from {username='slave'} at 127.0.0.1:33658
I20260929 14:47:45.221832 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7428351e1433462195d7432c7aa385d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:45 UTC) from {username='slave'} at 127.0.0.1:40458
I20260929 14:47:45.222451 5500 tablet_replica.cc:330] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: stopping tablet replica
I20260929 14:47:45.222678 5502 tablet_replica.cc:330] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9: stopping tablet replica
I20260929 14:47:45.222934 5500 raft_consensus.cc:2229] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:45.223107 5502 raft_consensus.cc:2229] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:45.223532 5500 raft_consensus.cc:2258] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:45.223572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5068
I20260929 14:47:45.223613 5502 raft_consensus.cc:2258] T 7428351e1433462195d7432c7aa385d3 P 0ea453b89fbe412cba0fb0acc9b45bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:45.226069 5501 tablet_replica.cc:330] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: stopping tablet replica
I20260929 14:47:45.226438 5501 raft_consensus.cc:2229] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:45.226804 5501 raft_consensus.cc:2258] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:45.226900 5500 ts_tablet_manager.cc:1905] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:45.228533 5501 ts_tablet_manager.cc:1905] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:45.229816 5500 ts_tablet_manager.cc:1918] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:45.229902 5500 log.cc:1196] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/7428351e1433462195d7432c7aa385d3
I20260929 14:47:45.230482 5500 ts_tablet_manager.cc:1939] T 7428351e1433462195d7432c7aa385d3 P 525da0e1e4c34c4397ad550e0c1374ab: Deleting consensus metadata
I20260929 14:47:45.231804 4992 catalog_manager.cc:4841] TS 525da0e1e4c34c4397ad550e0c1374ab (127.0.98.2:36827): tablet 7428351e1433462195d7432c7aa385d3 (table test-table [id=9a020542129e45f5874bae6ddc7a996e]) successfully deleted
I20260929 14:47:45.232491 5501 ts_tablet_manager.cc:1918] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:45.232589 5501 log.cc:1196] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/7428351e1433462195d7432c7aa385d3
I20260929 14:47:45.232913 5501 ts_tablet_manager.cc:1939] T 7428351e1433462195d7432c7aa385d3 P a3e99fa998d04b64b4c0509bb1f17ff5: Deleting consensus metadata
W20260929 14:47:45.233084 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42845 (error 108)
I20260929 14:47:45.233443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5199
I20260929 14:47:45.234998 4993 catalog_manager.cc:4841] TS a3e99fa998d04b64b4c0509bb1f17ff5 (127.0.98.3:36675): tablet 7428351e1433462195d7432c7aa385d3 (table test-table [id=9a020542129e45f5874bae6ddc7a996e]) successfully deleted
I20260929 14:47:45.235859 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.235472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42845 (local address 127.0.0.1:54048)
0929 14:47:45.235692 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:45.235773 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 14:47:45.236354 4995 catalog_manager.cc:4584] TS 0ea453b89fbe412cba0fb0acc9b45bc9 (127.0.98.1:42845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7428351e1433462195d7432c7aa385d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42845: connect: Connection refused (error 111)
I20260929 14:47:45.241505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5330
I20260929 14:47:45.246974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 4977
2026-09-29T14:47:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1214 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 14:47:45.269310 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:47:45.271230 392 webserver.cc:460] Webserver started at https://127.0.0.1:46185/ using document root <none> and password file <none>
I20260929 14:47:45.271445 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:47:45.271751 392 webserver.cc:460] Webserver started at http://127.0.0.1:45033/ using document root <none> and password file <none>
2026-09-29T14:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:45Z Disabled control of system clock
I20260929 14:47:45.283128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42003
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42003
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46185/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/testchainca.cert with env {}
W20260929 14:47:45.413077 5509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:45.413434 5509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:45.413478 5509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:45.417981 5509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:45.418066 5509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:45.418082 5509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:45.418100 5509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:45.418118 5509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:47:45.418134 5509 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:47:45.418187 5509 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46185/jwks.json
I20260929 14:47:45.424254 5509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46185/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:45.425642 5509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:45.426908 5509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:45.437093 5517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:45.437136 5514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:45.437177 5509 server_base.cc:1034] running on GCE node
W20260929 14:47:45.437136 5515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:45.438120 5509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:45.439453 5509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:45.440646 5509 hybrid_clock.cc:648] HybridClock initialized: now 1790693265440604 us; error 62 us; skew 500 ppm
I20260929 14:47:45.443025 5509 webserver.cc:460] Webserver started at http://127.0.98.62:37741/ using document root <none> and password file <none>
I20260929 14:47:45.443734 5509 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:45.443804 5509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:45.444052 5509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:45.446476 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "ac7333a8a208480da11010194919faec"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "0ff6f5cbdf71ab8c97cf6517c31405b3"
server_key_iv: "c05d29e5139586f3e28e4e60d9e78bc9"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.447130 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac7333a8a208480da11010194919faec"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "0ff6f5cbdf71ab8c97cf6517c31405b3"
server_key_iv: "c05d29e5139586f3e28e4e60d9e78bc9"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.451489 5509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:47:45.454555 5523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:45.456372 5509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 14:47:45.456557 5509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "ac7333a8a208480da11010194919faec"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "0ff6f5cbdf71ab8c97cf6517c31405b3"
server_key_iv: "c05d29e5139586f3e28e4e60d9e78bc9"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.456692 5509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:45.475620 5509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:45.476459 5509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:45.476691 5509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:45.485344 5509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42003
I20260929 14:47:45.485399 5575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42003 every 8 connection(s)
I20260929 14:47:45.486582 5509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:45.490284 5576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:45.491088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5509
I20260929 14:47:45.491217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:45.491536 392 external_mini_cluster.cc:1428] Setting key 25dcdfe1f55b81a6bde54f3de93e2f99
I20260929 14:47:45.496727 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec: Bootstrap starting.
I20260929 14:47:45.499292 5576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:45.500401 5576 log.cc:824] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:45.502823 5576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec: No bootstrap required, opened a new log
I20260929 14:47:45.503014 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.493242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.0.98.62:42003)
0929 14:47:45.493990 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.494009 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.494087 (+ 78us) server_negotiation.cc:407] Connection header received
0929 14:47:45.494899 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.494918 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.495406 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.495761 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:45.496257 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.497379 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.498629 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.499175 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.499311 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.500529 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:45.500538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:45.500576 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:45.501144 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:45.501228 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:45.501260 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:45.501381 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:45.501667 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:47:45.502042 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":182,"threads_started":1}
I20260929 14:47:45.506529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:47:45.507037 5576 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260929 14:47:45.507292 5576 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:45.507366 5576 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7333a8a208480da11010194919faec, State: Initialized, Role: FOLLOWER
I20260929 14:47:45.507903 5576 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260929 14:47:45.508073 5576 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:45.508201 5576 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:45.508312 5576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:45.509457 5576 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260929 14:47:45.509821 5576 leader_election.cc:302] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac7333a8a208480da11010194919faec; no voters:
I20260929 14:47:45.510183 5576 leader_election.cc:288] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:45.510362 5581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:45.510650 5581 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [term 1 LEADER]: Becoming Leader. State: Replica: ac7333a8a208480da11010194919faec, State: Running, Role: LEADER
I20260929 14:47:45.511129 5581 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } }
I20260929 14:47:45.511613 5576 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:45.513396 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac7333a8a208480da11010194919faec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } } }
I20260929 14:47:45.513427 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac7333a8a208480da11010194919faec. Latest consensus state: current_term: 1 leader_uuid: "ac7333a8a208480da11010194919faec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac7333a8a208480da11010194919faec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42003 } } }
I20260929 14:47:45.513572 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:45.513572 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:45.514225 5595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:45.517843 5595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:45.524806 5595 catalog_manager.cc:1347] Generated new cluster ID: 0c835c05fa8041319da9bd4534a4a1bc
I20260929 14:47:45.524919 5595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:45.538079 5595 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:45.539306 5595 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:45.544755 5595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac7333a8a208480da11010194919faec: Generated new TSK 0
I20260929 14:47:45.545658 5595 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:47:45.584146 5606 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 14:47:45.584386 5606 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:47:45.584933 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.565822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.0.98.62:42003)
0929 14:47:45.565996 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.566002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.566179 (+ 177us) server_negotiation.cc:407] Connection header received
0929 14:47:45.566451 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.566455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.566534 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.566710 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:47:45.567000 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.567856 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.568890 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.569286 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.569413 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.569871 (+ 458us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:47:45.584424 (+ 14553us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:47:45.584767 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 14:47:45.585053 5579 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57776: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 14:47:45.586444 5605 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:45.565655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42003 (local address 127.0.0.1:57776)
0929 14:47:45.566046 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:45.566063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:45.566236 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:45.566737 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:45.566740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:45.566752 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:47:45.566880 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:45.566885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.567999 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.568001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:45.568767 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:45.568775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.569428 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.569453 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:45.569566 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.569626 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:47:45.584741 (+ 15115us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:47:45.585054 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":146,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42003: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46185/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 14:47:45.587445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5509
2026-09-29T14:47:45Z chronyd exiting
I20260929 14:47:45.626806 392 test_util.cc:175] -----------------------------------------------
I20260929 14:47:45.626919 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790693238534145-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (359 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 14:47:45.628078 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:47:45.629065 392 webserver.cc:460] Webserver started at https://127.0.0.1:39975/ using document root <none> and password file <none>
I20260929 14:47:45.629159 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:47:45.629400 392 webserver.cc:460] Webserver started at http://127.0.0.1:35023/ using document root <none> and password file <none>
2026-09-29T14:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:45Z Disabled control of system clock
I20260929 14:47:45.639743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40715
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39975/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/testchainca.cert with env {}
W20260929 14:47:45.756942 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:45.757301 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:45.757354 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:45.761677 5612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:45.761759 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:45.761778 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:45.761799 5612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:45.761818 5612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:47:45.761838 5612 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:47:45.761854 5612 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39975/jwks.json
I20260929 14:47:45.767781 5612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39975/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:45.768972 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:45.770002 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:45.779773 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:45.779773 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:45.779826 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:45.780026 5612 server_base.cc:1034] running on GCE node
I20260929 14:47:45.780707 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:45.781885 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:45.783098 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790693265783082 us; error 40 us; skew 500 ppm
I20260929 14:47:45.785259 5612 webserver.cc:460] Webserver started at http://127.0.98.62:45503/ using document root <none> and password file <none>
I20260929 14:47:45.785907 5612 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:45.786000 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:45.786206 5612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:45.788429 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "c13533e0ba3a40c2ac0418adf221d3db"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "9e0d31617f60bee74a6a9be366dc2e68"
server_key_iv: "511e411c4d8965a259f0bc930e6192bd"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.789005 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "c13533e0ba3a40c2ac0418adf221d3db"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "9e0d31617f60bee74a6a9be366dc2e68"
server_key_iv: "511e411c4d8965a259f0bc930e6192bd"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.793159 5612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:47:45.796247 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:45.797875 5612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:45.798081 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "c13533e0ba3a40c2ac0418adf221d3db"
format_stamp: "Formatted at 2026-09-29 14:47:45 on dist-test-slave-5hd1"
server_key: "9e0d31617f60bee74a6a9be366dc2e68"
server_key_iv: "511e411c4d8965a259f0bc930e6192bd"
server_key_version: "encryptionkey@0"
I20260929 14:47:45.798223 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:45.817479 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:45.818202 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:45.818413 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:45.827231 5612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40715
I20260929 14:47:45.827270 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40715 every 8 connection(s)
I20260929 14:47:45.828430 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:45.832563 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:45.836223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5612
I20260929 14:47:45.836375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:45.836735 392 external_mini_cluster.cc:1428] Setting key b4271b4b554a94cd6040b1c94cf60442
I20260929 14:47:45.839995 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db: Bootstrap starting.
I20260929 14:47:45.842703 5679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:45.843607 5679 log.cc:824] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:45.846024 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db: No bootstrap required, opened a new log
I20260929 14:47:45.847822 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.838393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.98.62:40715)
0929 14:47:45.839371 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.839392 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.839438 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:47:45.840403 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.840424 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.840934 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.841309 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:45.841803 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.842843 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.843774 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.844381 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.844571 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.845696 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:45.845703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:45.845735 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:45.846245 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:45.846327 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:45.846351 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:45.846470 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:45.846655 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 14:47:45.846964 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":221,"threads_started":1}
I20260929 14:47:45.850607 5679 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260929 14:47:45.850911 5679 raft_consensus.cc:374] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:45.850984 5679 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13533e0ba3a40c2ac0418adf221d3db, State: Initialized, Role: FOLLOWER
I20260929 14:47:45.851284 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:47:45.851527 5679 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260929 14:47:45.851696 5679 raft_consensus.cc:388] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:45.851781 5679 raft_consensus.cc:482] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:45.851882 5679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:45.853122 5679 raft_consensus.cc:504] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260929 14:47:45.853482 5679 leader_election.cc:302] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13533e0ba3a40c2ac0418adf221d3db; no voters:
I20260929 14:47:45.853823 5679 leader_election.cc:288] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:45.854029 5684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:45.854416 5684 raft_consensus.cc:686] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [term 1 LEADER]: Becoming Leader. State: Replica: c13533e0ba3a40c2ac0418adf221d3db, State: Running, Role: LEADER
I20260929 14:47:45.854830 5684 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } }
I20260929 14:47:45.855240 5679 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:45.856657 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13533e0ba3a40c2ac0418adf221d3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } } }
I20260929 14:47:45.856791 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:45.857164 5686 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13533e0ba3a40c2ac0418adf221d3db. Latest consensus state: current_term: 1 leader_uuid: "c13533e0ba3a40c2ac0418adf221d3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13533e0ba3a40c2ac0418adf221d3db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40715 } } }
I20260929 14:47:45.857237 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:45.857290 5686 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:45.860638 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:45.866936 5693 catalog_manager.cc:1347] Generated new cluster ID: 8622a5e052734fb4b138f6c297a781f6
I20260929 14:47:45.867008 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:45.873584 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:45.874498 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:45.886744 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c13533e0ba3a40c2ac0418adf221d3db: Generated new TSK 0
I20260929 14:47:45.887916 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:47:45.917790 5709 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 14:47:45.917903 5709 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:47:45.918536 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:45.900446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.0.98.62:40715)
0929 14:47:45.900678 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:45.900686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:45.901033 (+ 347us) server_negotiation.cc:407] Connection header received
0929 14:47:45.901211 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:45.901213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:45.901271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:45.901378 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:47:45.901769 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.902468 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.903848 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.904131 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.904361 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.904699 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:47:45.918186 (+ 13487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39975/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:47:45.918422 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39975/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 14:47:45.918618 5682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39975/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 14:47:45.918967 5708 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:45.900379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40715 (local address 127.0.0.1:39102)
0929 14:47:45.900887 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:45.900905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:45.901063 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:45.901411 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:45.901414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:45.901423 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:47:45.901597 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:45.901604 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.902611 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.902614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:45.903705 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:45.903715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:45.904349 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:45.904374 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:45.904467 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:45.904499 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:47:45.918450 (+ 13951us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39975/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:47:45.918729 (+ 279us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39975/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":155,"threads_started":1}
I20260929 14:47:45.919744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5612
2026-09-29T14:47:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 14:47:45.960686 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:47:45.961726 392 webserver.cc:460] Webserver started at https://127.0.0.1:44565/ using document root <none> and password file <none>
I20260929 14:47:45.961846 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:47:45.962154 392 webserver.cc:460] Webserver started at http://127.0.0.1:40485/ using document root <none> and password file <none>
2026-09-29T14:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:45Z Disabled control of system clock
I20260929 14:47:45.973268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40083
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40083
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44565/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:47:46.107568 5715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:46.107906 5715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:46.107950 5715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:46.112571 5715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:46.112664 5715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:46.112681 5715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:46.112700 5715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:46.112716 5715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:47:46.112735 5715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:47:46.112748 5715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44565/jwks.json
I20260929 14:47:46.118144 5715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44565/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:46.119555 5715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:46.121450 5715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:47:46.131945 5715 server_base.cc:1034] running on GCE node
W20260929 14:47:46.131789 5720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:46.131789 5723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:46.131826 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:46.133021 5715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:46.134461 5715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:46.135701 5715 hybrid_clock.cc:648] HybridClock initialized: now 1790693266135661 us; error 52 us; skew 500 ppm
I20260929 14:47:46.151933 5715 webserver.cc:460] Webserver started at http://127.0.98.62:38427/ using document root <none> and password file <none>
I20260929 14:47:46.152877 5715 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:46.152997 5715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:46.153290 5715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:46.155969 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "911ff04f08b9484e94d7e0698db799fe"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "5b60dc7725c0d370fd128b11c74ba551"
server_key_iv: "aa812de8d971dd84d678bcdbbe39279e"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.156728 5715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "911ff04f08b9484e94d7e0698db799fe"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "5b60dc7725c0d370fd128b11c74ba551"
server_key_iv: "aa812de8d971dd84d678bcdbbe39279e"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.161427 5715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:47:46.164760 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:46.166332 5715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:46.166492 5715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "911ff04f08b9484e94d7e0698db799fe"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "5b60dc7725c0d370fd128b11c74ba551"
server_key_iv: "aa812de8d971dd84d678bcdbbe39279e"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.166635 5715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:46.178490 5715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:46.179333 5715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:46.179560 5715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:46.187913 5715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40083
I20260929 14:47:46.187942 5781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40083 every 8 connection(s)
I20260929 14:47:46.189091 5715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:46.191139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5715
I20260929 14:47:46.191298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:46.191628 392 external_mini_cluster.cc:1428] Setting key 714af65d0feaf95ad738a13bed618f7b
I20260929 14:47:46.192991 5782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:46.200543 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe: Bootstrap starting.
I20260929 14:47:46.202814 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:46.193349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.98.62:40083)
0929 14:47:46.194062 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:46.194080 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:46.194127 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:47:46.194905 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:46.194924 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:46.195376 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:46.195711 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:46.196268 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.197349 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.198702 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.199247 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.199377 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.200493 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:46.200498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:46.200514 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:46.201060 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:46.201138 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:46.201161 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:46.201248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:46.201578 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 14:47:46.201922 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":169,"threads_started":1}
I20260929 14:47:46.203279 5782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:46.204108 5782 log.cc:824] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:46.206305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:47:46.206467 5782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe: No bootstrap required, opened a new log
I20260929 14:47:46.210264 5782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } }
I20260929 14:47:46.210572 5782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:46.210654 5782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911ff04f08b9484e94d7e0698db799fe, State: Initialized, Role: FOLLOWER
I20260929 14:47:46.211095 5782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } }
I20260929 14:47:46.211243 5782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:46.211334 5782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:46.211450 5782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:46.212473 5782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } }
I20260929 14:47:46.212838 5782 leader_election.cc:302] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 911ff04f08b9484e94d7e0698db799fe; no voters:
I20260929 14:47:46.213191 5782 leader_election.cc:288] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:46.213414 5787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:46.213835 5787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [term 1 LEADER]: Becoming Leader. State: Replica: 911ff04f08b9484e94d7e0698db799fe, State: Running, Role: LEADER
I20260929 14:47:46.214305 5787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } }
I20260929 14:47:46.214643 5782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:46.216622 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "911ff04f08b9484e94d7e0698db799fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } } }
I20260929 14:47:46.216661 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 911ff04f08b9484e94d7e0698db799fe. Latest consensus state: current_term: 1 leader_uuid: "911ff04f08b9484e94d7e0698db799fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "911ff04f08b9484e94d7e0698db799fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40083 } } }
I20260929 14:47:46.216833 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:46.216833 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:46.217526 5802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:46.221354 5802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:46.227172 5802 catalog_manager.cc:1347] Generated new cluster ID: 5e0933ac702448cf8ff5d0c236c3541a
I20260929 14:47:46.227263 5802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:46.239516 5802 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:46.241163 5802 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:46.248060 5802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 911ff04f08b9484e94d7e0698db799fe: Generated new TSK 0
I20260929 14:47:46.249089 5802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:47:46.264029 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:46.258430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.98.62:40083)
0929 14:47:46.258596 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:46.258602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:46.258831 (+ 229us) server_negotiation.cc:407] Connection header received
0929 14:47:46.259036 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:46.259038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:46.259095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:46.259232 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:47:46.259495 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.260045 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.261337 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.261557 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.261636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.261997 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:47:46.263602 (+ 1605us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:47:46.263877 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 14:47:46.264128 5785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48330: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 14:47:46.267246 5811 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:46.258350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40083 (local address 127.0.0.1:48330)
0929 14:47:46.258716 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:46.258734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:46.258874 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:46.259230 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:46.259234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:46.259243 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:47:46.259383 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:46.259390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.260155 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.260159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:46.261205 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:46.261215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.261657 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.261683 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:46.261771 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.261802 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:47:46.263791 (+ 1989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:47:46.267067 (+ 3276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40083: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":2968,"thread_start_us":132,"threads_started":1}
I20260929 14:47:46.267740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5715
E20260929 14:47:46.278723 5812 webserver.cc:533] Webserver: sslize failed:
E20260929 14:47:46.278820 5812 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T14:47:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T14:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:46Z Disabled control of system clock
I20260929 14:47:46.319389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34973
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34973
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/testchainca.cert with env {}
W20260929 14:47:46.448593 5816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:46.448906 5816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:46.448951 5816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:46.453284 5816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:46.453358 5816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:46.453373 5816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:46.453392 5816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:46.453408 5816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:47:46.453474 5816 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:47:46.453493 5816 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 14:47:46.459667 5816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:46.461171 5816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:46.462476 5816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:46.472810 5821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:46.472810 5822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:46.472865 5824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:46.473280 5816 server_base.cc:1034] running on GCE node
I20260929 14:47:46.474004 5816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:46.475301 5816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:46.476526 5816 hybrid_clock.cc:648] HybridClock initialized: now 1790693266476502 us; error 48 us; skew 500 ppm
I20260929 14:47:46.478677 5816 webserver.cc:460] Webserver started at http://127.0.98.62:39727/ using document root <none> and password file <none>
I20260929 14:47:46.479357 5816 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:46.479455 5816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:46.479703 5816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:46.481971 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "774eb78bd1c6417cb82141d01b3feeda"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "8b46fcdeb79c71d89b3ab8cde3df6d98"
server_key_iv: "3f2ad7cc9c4e14d081f59a0dc471b3dc"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.482585 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "774eb78bd1c6417cb82141d01b3feeda"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "8b46fcdeb79c71d89b3ab8cde3df6d98"
server_key_iv: "3f2ad7cc9c4e14d081f59a0dc471b3dc"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.486886 5816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 14:47:46.489789 5830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:46.491328 5816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:46.491484 5816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "774eb78bd1c6417cb82141d01b3feeda"
format_stamp: "Formatted at 2026-09-29 14:47:46 on dist-test-slave-5hd1"
server_key: "8b46fcdeb79c71d89b3ab8cde3df6d98"
server_key_iv: "3f2ad7cc9c4e14d081f59a0dc471b3dc"
server_key_version: "encryptionkey@0"
I20260929 14:47:46.491611 5816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:46.527207 5816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:46.528012 5816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:46.528266 5816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:46.536015 5816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34973
I20260929 14:47:46.536038 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34973 every 8 connection(s)
I20260929 14:47:46.537160 5816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:46.538014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5816
I20260929 14:47:46.538174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:46.538554 392 external_mini_cluster.cc:1428] Setting key a16cd6f49db65bf2b11092e7c9f547b2
I20260929 14:47:46.541759 5883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:46.549031 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda: Bootstrap starting.
I20260929 14:47:46.549851 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:46.540318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.0.98.62:34973)
0929 14:47:46.541205 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:46.541222 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:46.541273 (+ 51us) server_negotiation.cc:407] Connection header received
0929 14:47:46.542107 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:46.542130 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:46.542738 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:46.543180 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:46.543682 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.544841 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.545779 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.546338 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.546536 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.547632 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:46.547638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:46.547652 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:47:46.548095 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:46.548155 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:47:46.548268 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:46.548381 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:46.548553 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:47:46.548853 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":203,"threads_started":1}
I20260929 14:47:46.552531 5883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:46.553613 5883 log.cc:824] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:46.554288 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:47:46.556578 5883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda: No bootstrap required, opened a new log
I20260929 14:47:46.561749 5883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } }
I20260929 14:47:46.562134 5883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:46.562206 5883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774eb78bd1c6417cb82141d01b3feeda, State: Initialized, Role: FOLLOWER
I20260929 14:47:46.562788 5883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } }
I20260929 14:47:46.562932 5883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:46.563045 5883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:46.563160 5883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:46.564369 5883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } }
I20260929 14:47:46.564832 5883 leader_election.cc:302] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774eb78bd1c6417cb82141d01b3feeda; no voters:
I20260929 14:47:46.565217 5883 leader_election.cc:288] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:46.565402 5888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:46.565742 5888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [term 1 LEADER]: Becoming Leader. State: Replica: 774eb78bd1c6417cb82141d01b3feeda, State: Running, Role: LEADER
I20260929 14:47:46.566143 5888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } }
I20260929 14:47:46.566733 5883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:46.568146 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774eb78bd1c6417cb82141d01b3feeda. Latest consensus state: current_term: 1 leader_uuid: "774eb78bd1c6417cb82141d01b3feeda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } } }
I20260929 14:47:46.568332 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:46.568761 5897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:46.568723 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774eb78bd1c6417cb82141d01b3feeda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb78bd1c6417cb82141d01b3feeda" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34973 } } }
I20260929 14:47:46.568917 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:46.572083 5897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:46.578671 5897 catalog_manager.cc:1347] Generated new cluster ID: fbd0ae60c92f48608b316b5112930219
I20260929 14:47:46.578769 5897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:46.590865 5897 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:46.592370 5897 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:46.599529 5897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774eb78bd1c6417cb82141d01b3feeda: Generated new TSK 0
I20260929 14:47:46.600435 5897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:47:46.702576 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:46.610793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.0.98.62:34973)
0929 14:47:46.610970 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:46.610975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:46.611241 (+ 266us) server_negotiation.cc:407] Connection header received
0929 14:47:46.611423 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:46.611425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:46.611477 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:46.611584 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:47:46.611857 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.612515 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.613789 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.614000 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.614079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.614429 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:47:46.701893 (+ 87464us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:47:46.702332 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 14:47:46.702692 5886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38962: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 14:47:46.702757 5912 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:46.610746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34973 (local address 127.0.0.1:38962)
0929 14:47:46.611124 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:46.611141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:46.611272 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:46.611615 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:46.611619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:46.611627 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:47:46.611741 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:46.611748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.612636 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.612640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:46.613661 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:46.613671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:46.614099 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:46.614125 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:46.614219 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:46.614248 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:47:46.702240 (+ 87992us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:47:46.702507 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":163,"threads_started":1}
I20260929 14:47:46.703651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5816
2026-09-29T14:47:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (415 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (139 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (142 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:47 dist-test-slave-5hd1 krb5kdc[5924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:47 dist-test-slave-5hd1 krb5kdc[5924](info): set up 2 sockets
Sep 29 14:47:47 dist-test-slave-5hd1 krb5kdc[5924](info): commencing operation
krb5kdc: starting...
W20260929 14:47:48.048633 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:48.197405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:48.319077 5937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:48.319394 5937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:48.319479 5937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:48.323644 5937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:48.323751 5937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:48.323786 5937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:48.323827 5937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:48.323866 5937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:48.329576 5937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:48.330763 5937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:48.331840 5937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:48.340978 5943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:48.340970 5945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:48.341034 5942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:48.341652 5937 server_base.cc:1034] running on GCE node
I20260929 14:47:48.342383 5937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:48.343771 5937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:48.344980 5937 hybrid_clock.cc:648] HybridClock initialized: now 1790693268344956 us; error 38 us; skew 500 ppm
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:48.348302 5937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:47:48.349403 5937 webserver.cc:460] Webserver started at http://127.0.98.62:46243/ using document root <none> and password file <none>
I20260929 14:47:48.350073 5937 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:48.350139 5937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:48.350334 5937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:48.352818 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "a5fbd0d49f66433ea3765ef0c9374b5f"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "3ecc1bf21a4a22afdd8250b18234c2d1"
server_key_iv: "c1fd9c47ae17ec41032ef225ba455522"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.353408 5937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5fbd0d49f66433ea3765ef0c9374b5f"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "3ecc1bf21a4a22afdd8250b18234c2d1"
server_key_iv: "c1fd9c47ae17ec41032ef225ba455522"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.358222 5937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:47:48.361550 5952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.363328 5937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 14:47:48.363477 5937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "a5fbd0d49f66433ea3765ef0c9374b5f"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "3ecc1bf21a4a22afdd8250b18234c2d1"
server_key_iv: "c1fd9c47ae17ec41032ef225ba455522"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.363603 5937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:48.381294 5937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:48.385162 5937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:48.385435 5937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:48.394315 5937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44417
I20260929 14:47:48.394336 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44417 every 8 connection(s)
I20260929 14:47:48.395576 5937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:48.398788 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:48.405133 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: Bootstrap starting.
I20260929 14:47:48.405871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 5937
I20260929 14:47:48.406025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:48.406405 392 external_mini_cluster.cc:1428] Setting key 14e631d830600885f7a87a9ba81ee8fb
I20260929 14:47:48.408342 6005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:48.409260 6005 log.cc:824] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:48.411655 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: No bootstrap required, opened a new log
I20260929 14:47:48.415724 6005 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260929 14:47:48.415966 6005 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:48.415999 6005 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fbd0d49f66433ea3765ef0c9374b5f, State: Initialized, Role: FOLLOWER
I20260929 14:47:48.416567 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260929 14:47:48.416738 6005 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:48.416807 6005 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:48.416927 6005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:48.418036 6005 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260929 14:47:48.418434 6005 leader_election.cc:302] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5fbd0d49f66433ea3765ef0c9374b5f; no voters:
I20260929 14:47:48.418857 6005 leader_election.cc:288] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:48.419107 6010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:48.419488 6010 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [term 1 LEADER]: Becoming Leader. State: Replica: a5fbd0d49f66433ea3765ef0c9374b5f, State: Running, Role: LEADER
I20260929 14:47:48.419911 6010 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } }
I20260929 14:47:48.420424 6005 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:48.421639 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fbd0d49f66433ea3765ef0c9374b5f. Latest consensus state: current_term: 1 leader_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } } }
I20260929 14:47:48.421784 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:48.422080 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5fbd0d49f66433ea3765ef0c9374b5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44417 } } }
I20260929 14:47:48.422225 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:48.424760 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:48.407885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.98.62:44417)
0929 14:47:48.408577 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:48.408601 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:48.408655 (+ 54us) server_negotiation.cc:407] Connection header received
0929 14:47:48.409502 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:48.409526 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:48.409924 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:48.410269 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:48.411060 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.412564 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.413534 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.414248 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.414465 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:48.417758 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:48.417765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:48.417782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:48.417824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:48.420258 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:48.420904 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:48.420910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:48.420917 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:48.421063 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:48.421429 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:48.421434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:48.421436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:48.422372 (+ 936us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:48.422671 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:48.423258 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 14:47:48.423631 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":234,"threads_started":1}
W20260929 14:47:48.426090 6025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:48.426186 6025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:48.426254 6019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:48.426939 6019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:48.433396 6019 catalog_manager.cc:1347] Generated new cluster ID: 12e8b38e37ec4ef997e2938fc2a79a30
I20260929 14:47:48.433488 6019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:48.474728 6019 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:48.476280 6019 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:48.491421 6019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5fbd0d49f66433ea3765ef0c9374b5f: Generated new TSK 0
I20260929 14:47:48.492388 6019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:47:48.535261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:48.668402 6031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:48.668733 6031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:48.668784 6031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:48.673306 6031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:48.673386 6031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:48.673453 6031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:48.679270 6031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:48.680778 6031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:48.682087 6031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:48.693787 6039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:48.693787 6036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:48.693985 6031 server_base.cc:1034] running on GCE node
W20260929 14:47:48.694110 6037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:48.694662 6031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:48.695482 6031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:48.696676 6031 hybrid_clock.cc:648] HybridClock initialized: now 1790693268696663 us; error 37 us; skew 500 ppm
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:48.699702 6031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:47:48.700729 6031 webserver.cc:460] Webserver started at http://127.0.98.1:46681/ using document root <none> and password file <none>
I20260929 14:47:48.701349 6031 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:48.701411 6031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:48.701593 6031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:48.703828 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbe8b52d42a14218b6b2a3e40fe6cbd7"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "44463e80c72ddaf849a37ecc1450d013"
server_key_iv: "ad77dad73687abe312cc467dedfa16b7"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.704442 6031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbe8b52d42a14218b6b2a3e40fe6cbd7"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "44463e80c72ddaf849a37ecc1450d013"
server_key_iv: "ad77dad73687abe312cc467dedfa16b7"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.708914 6031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:47:48.711970 6046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.713761 6031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:48.713902 6031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "dbe8b52d42a14218b6b2a3e40fe6cbd7"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "44463e80c72ddaf849a37ecc1450d013"
server_key_iv: "ad77dad73687abe312cc467dedfa16b7"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.714035 6031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:48.734211 6031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:48.737545 6031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:48.737758 6031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:48.738389 6031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:48.739257 6031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:48.739306 6031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.739373 6031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:48.739411 6031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.749393 6031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43245
I20260929 14:47:48.749475 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43245 every 8 connection(s)
I20260929 14:47:48.750417 6031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:48.753923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6031
I20260929 14:47:48.754053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:48.754369 392 external_mini_cluster.cc:1428] Setting key 6e6c14aaed07f0d2638954e63e7afa39
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:48.766990 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:48.752772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34083 (local address 127.0.98.62:44417)
0929 14:47:48.752967 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:48.752972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:48.753882 (+ 910us) server_negotiation.cc:407] Connection header received
0929 14:47:48.755690 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:48.755694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:48.755751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:48.755832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:48.757289 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.758202 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.759568 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.759992 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.760126 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:48.762765 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:48.762770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:48.762775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:48.762807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:48.764645 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:48.765242 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:48.765245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:48.765247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:48.765322 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:48.765645 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:48.765648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:48.765649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:48.765889 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:48.766001 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:48.766653 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 14:47:48.766794 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:48.768079 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:48.753138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.1:34083)
0929 14:47:48.753688 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:48.753742 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:48.755456 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:48.756033 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:48.756047 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:48.756556 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:48.757134 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:48.757148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.758399 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.758404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:48.759401 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:48.759414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:48.760144 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:48.760147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:48.760437 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:48.761080 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:48.761096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:48.762568 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:48.764825 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:48.764831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:48.764849 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:48.765136 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:48.765425 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:48.765427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:48.765428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:48.765557 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:48.766013 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:48.766024 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:48.766382 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 14:47:48.766616 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 14:47:48.767023 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20260929 14:47:48.769646 6160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260929 14:47:48.769946 6160 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:48.770573 6160 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
I20260929 14:47:48.772629 5969 ts_manager.cc:194] Registered new tserver with Master: dbe8b52d42a14218b6b2a3e40fe6cbd7 (127.0.98.1:43245)
I20260929 14:47:48.774470 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:47:48.788246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:48.907603 6165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:48.907889 6165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:48.907931 6165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:48.912250 6165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:48.912319 6165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:48.912413 6165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:48.917768 6165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:48.918956 6165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:48.920029 6165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:48.930757 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:48.930757 6170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:48.930886 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:48.931315 6165 server_base.cc:1034] running on GCE node
I20260929 14:47:48.931707 6165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:48.932384 6165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:48.933589 6165 hybrid_clock.cc:648] HybridClock initialized: now 1790693268933540 us; error 74 us; skew 500 ppm
Sep 29 14:47:48 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:48.936735 6165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:47:48.937664 6165 webserver.cc:460] Webserver started at http://127.0.98.2:42191/ using document root <none> and password file <none>
I20260929 14:47:48.938289 6165 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:48.938391 6165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:48.938638 6165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:48.940876 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd58ecd5b4014e51b47907a8bdbbf000"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "b0650d8b44ce53e9af017728d937d1fa"
server_key_iv: "1ef96a88ca099110f29c5d89da6e4c50"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.941499 6165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd58ecd5b4014e51b47907a8bdbbf000"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "b0650d8b44ce53e9af017728d937d1fa"
server_key_iv: "1ef96a88ca099110f29c5d89da6e4c50"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.945766 6165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:48.948607 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.950052 6165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:48.950196 6165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "cd58ecd5b4014e51b47907a8bdbbf000"
format_stamp: "Formatted at 2026-09-29 14:47:48 on dist-test-slave-5hd1"
server_key: "b0650d8b44ce53e9af017728d937d1fa"
server_key_iv: "1ef96a88ca099110f29c5d89da6e4c50"
server_key_version: "encryptionkey@0"
I20260929 14:47:48.950322 6165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:48.982497 6165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:48.985828 6165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:48.986085 6165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:48.986707 6165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:48.987567 6165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:48.987643 6165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.987715 6165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:48.987761 6165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:48.998279 6165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35967
I20260929 14:47:48.998333 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35967 every 8 connection(s)
I20260929 14:47:48.999399 6165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:49.006354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6165
I20260929 14:47:49.006500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:49.006837 392 external_mini_cluster.cc:1428] Setting key 9a4f27a16ee479c3852b5d02f31dfbd0
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:49.018111 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.001987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55923 (local address 127.0.98.62:44417)
0929 14:47:49.002215 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.002221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.003188 (+ 967us) server_negotiation.cc:407] Connection header received
0929 14:47:49.004953 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.004956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.005018 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.005119 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.007033 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.007697 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.009643 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.009873 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.009973 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.013574 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:49.013578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:49.013580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:49.013606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:49.015202 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.016036 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.016038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.016040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.016115 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.016654 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.016656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.016658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.016847 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:49.016926 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:49.017862 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 14:47:49.017979 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:47:49.020032 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.002384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.2:55923)
0929 14:47:49.002946 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:49.003019 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:49.004676 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:49.005357 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:49.005366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:49.005966 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:49.006831 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:49.006855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.007876 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.007880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:49.009476 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:49.009491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.010070 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.010078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:49.010347 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.011314 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:49.011334 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:49.013336 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:49.015357 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:49.015368 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:49.015400 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:49.015892 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:49.016270 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:49.016275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:49.016279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:49.016531 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:49.016968 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:49.016982 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:49.017515 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 14:47:49.017796 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 14:47:49.018299 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":121,"threads_started":1}
I20260929 14:47:49.021857 6294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260929 14:47:49.022202 6294 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:49.022837 6294 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:47:49.024387 5969 ts_manager.cc:194] Registered new tserver with Master: cd58ecd5b4014e51b47907a8bdbbf000 (127.0.98.2:35967)
I20260929 14:47:49.025700 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:47:49.044632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:49.178884 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:49.179240 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:49.179289 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:49.183699 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:49.183797 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:49.183912 6299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:49.190059 6299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:49.191615 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:49.193025 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:49.204684 6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:49.204800 6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:49.204684 6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:49.204979 6299 server_base.cc:1034] running on GCE node
I20260929 14:47:49.205684 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:49.206466 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:49.207677 6299 hybrid_clock.cc:648] HybridClock initialized: now 1790693269207654 us; error 38 us; skew 500 ppm
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.211364 6299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:47:49.212718 6299 webserver.cc:460] Webserver started at http://127.0.98.3:40895/ using document root <none> and password file <none>
I20260929 14:47:49.213426 6299 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:49.213503 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:49.213729 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:49.216331 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "a673592271e94ad79f4c3dd2637878b2"
format_stamp: "Formatted at 2026-09-29 14:47:49 on dist-test-slave-5hd1"
server_key: "87b8036dd9d4407138f2b2933de217de"
server_key_iv: "2d76854723e6a440f6451c1c78520f5f"
server_key_version: "encryptionkey@0"
I20260929 14:47:49.216971 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a673592271e94ad79f4c3dd2637878b2"
format_stamp: "Formatted at 2026-09-29 14:47:49 on dist-test-slave-5hd1"
server_key: "87b8036dd9d4407138f2b2933de217de"
server_key_iv: "2d76854723e6a440f6451c1c78520f5f"
server_key_version: "encryptionkey@0"
I20260929 14:47:49.222183 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:47:49.225471 6314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:49.227181 6299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:49.227360 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "a673592271e94ad79f4c3dd2637878b2"
format_stamp: "Formatted at 2026-09-29 14:47:49 on dist-test-slave-5hd1"
server_key: "87b8036dd9d4407138f2b2933de217de"
server_key_iv: "2d76854723e6a440f6451c1c78520f5f"
server_key_version: "encryptionkey@0"
I20260929 14:47:49.227521 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:49.241720 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:49.245369 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:49.245591 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:49.246279 6299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:49.247218 6299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:49.247288 6299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:49.247364 6299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:49.247417 6299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:49.259318 6299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40175
I20260929 14:47:49.259413 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40175 every 8 connection(s)
I20260929 14:47:49.260516 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:49.262880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6299
I20260929 14:47:49.263039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:49.263365 392 external_mini_cluster.cc:1428] Setting key ad922947f3fe6a5b12d898b917c83df4
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:47:49.277330 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.262897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34285 (local address 127.0.98.62:44417)
0929 14:47:49.263071 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.263075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.264256 (+ 1181us) server_negotiation.cc:407] Connection header received
0929 14:47:49.265974 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.265978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.266089 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.266176 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.267592 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.268375 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.269785 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.270051 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.270149 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.273244 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:49.273248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:49.273254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:49.273283 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:49.274694 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.275293 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.275298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.275300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.275388 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.275764 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.275769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.275771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.275997 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:49.276115 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:49.276931 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 14:47:49.277118 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:47:49.278466 6430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.263475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44417 (local address 127.0.98.3:34285)
0929 14:47:49.264066 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:49.264124 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:49.265619 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:49.266377 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:49.266384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:49.266792 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:49.267415 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:49.267429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.268545 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.268577 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:49.269643 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:49.269655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.270220 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.270224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:49.270376 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.271056 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:49.271073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:49.273006 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:49.274835 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:49.274843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:49.274873 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:49.275184 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:49.275490 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:49.275493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:49.275495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:49.275664 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:49.276132 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:49.276140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:49.276523 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 14:47:49.276777 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 14:47:49.277208 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":191,"threads_started":1}
I20260929 14:47:49.279820 6428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44417
I20260929 14:47:49.280077 6428 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:49.280675 6428 heartbeater.cc:511] Master 127.0.98.62:44417 requested a full tablet report, sending...
I20260929 14:47:49.281898 5969 ts_manager.cc:194] Registered new tserver with Master: a673592271e94ad79f4c3dd2637878b2 (127.0.98.3:40175)
I20260929 14:47:49.282655 5969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34285
I20260929 14:47:49.291282 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/test-admin.keytab.
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.310973 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 14:47:49.314391 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:49.312803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.0.98.1:43245)
0929 14:47:49.313180 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.313187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.313216 (+ 29us) server_negotiation.cc:407] Connection header received
0929 14:47:49.313283 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.313288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.313488 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.313632 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.314201 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56056: BlockingRecv error: recv got EOF from 127.0.0.1:56056 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":133,"threads_started":1}
W20260929 14:47:49.314488 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.316056 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.317198 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:49.328795 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.318355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.0.98.1:43245)
0929 14:47:49.318502 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.318508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.318609 (+ 101us) server_negotiation.cc:407] Connection header received
0929 14:47:49.318813 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.318816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.318867 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.318961 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.319529 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.320469 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.321401 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.321931 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.322053 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.324752 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:49.324761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:49.324780 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:49.324806 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:49.326517 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.327015 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.327021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.327022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.327108 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.327530 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.327537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.327541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.327901 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:49.328104 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:49.328473 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 14:47:49.328646 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 14:47:49.333976 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:49.332562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.0.98.1:43245)
0929 14:47:49.332734 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.332738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.332899 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:47:49.333043 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.333045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.333095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.333176 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.333857 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56082: BlockingRecv error: recv got EOF from 127.0.0.1:56082 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 14:47:49.334151 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.335410 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.336454 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:49.347301 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.338595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.0.98.1:43245)
0929 14:47:49.338774 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.338780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.338915 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:47:49.339101 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.339103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.339147 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.339222 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.339742 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.340340 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.341234 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.341463 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.341551 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.344211 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:49.344215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:49.344218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:49.344248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:49.345637 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.346085 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.346091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.346093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.346173 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.346454 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.346457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.346459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.346631 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:49.346789 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:49.347021 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:47:49.347133 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260929 14:47:49.350436 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:49.349361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.0.98.1:43245)
0929 14:47:49.349506 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.349511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.349645 (+ 134us) server_negotiation.cc:407] Connection header received
0929 14:47:49.349794 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.349797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.349837 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.349937 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.350327 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56100: BlockingRecv error: recv got EOF from 127.0.0.1:56100 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 14:47:49.350600 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790693238534145-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.351706 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:49.352649 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[5924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:47:49.362735 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:49.353641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.1:43245)
0929 14:47:49.353770 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:49.353775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:49.353890 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:47:49.354041 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:49.354044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:49.354086 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:49.354176 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:49.354676 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.355286 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.356094 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:49.356402 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:49.356510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:49.359157 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:49.359161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:49.359163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:49.359189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:49.360981 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.361433 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.361439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.361441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.361526 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:49.361838 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:49.361843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:49.361845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:49.362039 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:49.362207 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:49.362446 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 14:47:49.362553 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:47:49.364249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6031
I20260929 14:47:49.371276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6165
I20260929 14:47:49.378006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6299
I20260929 14:47:49.383617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 5937
2026-09-29T14:47:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2401 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[6448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[6448](info): set up 2 sockets
Sep 29 14:47:49 dist-test-slave-5hd1 krb5kdc[6448](info): commencing operation
krb5kdc: starting...
W20260929 14:47:50.440727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:50 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:47:50.584745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:50.703996 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:50.704392 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:50.704478 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:50.708688 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:50.708789 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:50.708828 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:50.708866 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:50.708902 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:50.714516 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:50.715663 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:50.716766 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:50.726236 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:50.726229 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:50.726780 6461 server_base.cc:1034] running on GCE node
W20260929 14:47:50.726847 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:50.727591 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:50.728868 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:50.730095 6461 hybrid_clock.cc:648] HybridClock initialized: now 1790693270730073 us; error 42 us; skew 500 ppm
Sep 29 14:47:50 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:50.733189 6461 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:47:50.734239 6461 webserver.cc:460] Webserver started at http://127.0.98.62:37215/ using document root <none> and password file <none>
I20260929 14:47:50.734824 6461 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:50.734887 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:50.735069 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:50.737303 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "e2c48ccb36f94260a3b4ed461872d21e"
format_stamp: "Formatted at 2026-09-29 14:47:50 on dist-test-slave-5hd1"
server_key: "fcf48f8f6326c708098b6068866effc7"
server_key_iv: "9b26d8bf27ab4716efcdfe853f65d77a"
server_key_version: "encryptionkey@0"
I20260929 14:47:50.737836 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2c48ccb36f94260a3b4ed461872d21e"
format_stamp: "Formatted at 2026-09-29 14:47:50 on dist-test-slave-5hd1"
server_key: "fcf48f8f6326c708098b6068866effc7"
server_key_iv: "9b26d8bf27ab4716efcdfe853f65d77a"
server_key_version: "encryptionkey@0"
I20260929 14:47:50.742056 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:50.744925 6476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:50.746449 6461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:50.746582 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "e2c48ccb36f94260a3b4ed461872d21e"
format_stamp: "Formatted at 2026-09-29 14:47:50 on dist-test-slave-5hd1"
server_key: "fcf48f8f6326c708098b6068866effc7"
server_key_iv: "9b26d8bf27ab4716efcdfe853f65d77a"
server_key_version: "encryptionkey@0"
I20260929 14:47:50.746680 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:50.763224 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:50.767041 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:50.767217 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:50.775050 6461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40107
I20260929 14:47:50.775513 6528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40107 every 8 connection(s)
I20260929 14:47:50.776105 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:50.779644 6529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:50.782789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6461
I20260929 14:47:50.782972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:50.783345 392 external_mini_cluster.cc:1428] Setting key d6dea5a5490ced2223a14a42ac44d5ed
I20260929 14:47:50.787141 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Bootstrap starting.
I20260929 14:47:50.790400 6529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:50.791446 6529 log.cc:824] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:47:50 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:50.794225 6529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: No bootstrap required, opened a new log
I20260929 14:47:50.798400 6529 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } }
I20260929 14:47:50.798771 6529 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:50.798869 6529 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c48ccb36f94260a3b4ed461872d21e, State: Initialized, Role: FOLLOWER
I20260929 14:47:50.799423 6529 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } }
I20260929 14:47:50.799592 6529 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:50.799700 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:50.785086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.0.98.62:40107)
0929 14:47:50.785552 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:50.785568 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:50.785607 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:47:50.786411 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:50.786429 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:50.786796 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:50.787125 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:50.788241 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:50.789341 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:50.790628 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:50.791138 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:50.791279 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:50.794405 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:50.794411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:50.794426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:50.794455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:50.796313 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:50.796790 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:50.796796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:50.796809 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:50.796921 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:50.797217 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:50.797220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:50.797222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:50.797720 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:50.797961 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:50.798210 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 14:47:50.798544 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":154,"threads_started":1}
I20260929 14:47:50.799690 6529 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:50.799943 6529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:50.801182 6529 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } }
I20260929 14:47:50.801596 6529 leader_election.cc:302] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c48ccb36f94260a3b4ed461872d21e; no voters:
I20260929 14:47:50.802035 6529 leader_election.cc:288] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:50.802176 6534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:50.802395 6534 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 LEADER]: Becoming Leader. State: Replica: e2c48ccb36f94260a3b4ed461872d21e, State: Running, Role: LEADER
I20260929 14:47:50.802811 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } }
I20260929 14:47:50.803613 6529 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:50.804801 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } }
I20260929 14:47:50.805022 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:50.805390 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c48ccb36f94260a3b4ed461872d21e. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } }
I20260929 14:47:50.805490 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:50.805965 6543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:50.809547 6543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:50.816394 6543 catalog_manager.cc:1347] Generated new cluster ID: 2f615b46235e411b87ca818eb10de484
I20260929 14:47:50.816495 6543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:50.829319 6543 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:50.830803 6543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:50.852067 6543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Generated new TSK 0
I20260929 14:47:50.853075 6543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:47:50.887653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:51.018922 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:51.019414 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:51.019486 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:51.024093 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:51.024212 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:51.024286 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:51.030580 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:51.031991 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:51.033247 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:51.044946 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.044963 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.045102 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:51.044987 6555 server_base.cc:1034] running on GCE node
I20260929 14:47:51.045832 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:51.046624 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:51.047842 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790693271047825 us; error 39 us; skew 500 ppm
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:51.051510 6555 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:47:51.052742 6555 webserver.cc:460] Webserver started at http://127.0.98.1:36679/ using document root <none> and password file <none>
I20260929 14:47:51.053440 6555 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:51.053537 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:51.053807 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:51.056324 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "8681f175c4e549a487c8eda1ed0a934d"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "b92ee93dd73aaa917bd2d0bcc466af9d"
server_key_iv: "a5e5e78585169a7417603374b838d7c9"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.056964 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8681f175c4e549a487c8eda1ed0a934d"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "b92ee93dd73aaa917bd2d0bcc466af9d"
server_key_iv: "a5e5e78585169a7417603374b838d7c9"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.061311 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:51.064105 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.065653 6555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:51.065799 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "8681f175c4e549a487c8eda1ed0a934d"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "b92ee93dd73aaa917bd2d0bcc466af9d"
server_key_iv: "a5e5e78585169a7417603374b838d7c9"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.065917 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:51.078552 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:51.083734 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:51.083983 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:51.084923 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:51.086308 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:51.086385 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.086457 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:51.086491 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.098237 6555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42085
I20260929 14:47:51.098351 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42085 every 8 connection(s)
I20260929 14:47:51.099402 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:51.105892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6555
I20260929 14:47:51.106042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:51.106387 392 external_mini_cluster.cc:1428] Setting key 9304c317fd1080bb51f8fa96ee4c85b7
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:51.117168 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.101681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34511 (local address 127.0.98.62:40107)
0929 14:47:51.101854 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.101858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.102885 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 14:47:51.104572 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.104576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.104627 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.104740 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.106274 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.107039 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.108510 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.108760 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.108852 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.112306 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.112310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.112315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.112348 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.114276 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.115101 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.115104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.115106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.115228 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.115726 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.115729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.115730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.115934 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.116027 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.116905 (+ 878us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.117041 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:47:51.118718 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.102133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:34511)
0929 14:47:51.102705 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.102758 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.104299 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.104937 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.104944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.105376 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.106080 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.106101 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.107180 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.107184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.108355 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.108366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.108932 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.108939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.109178 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.109988 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.110016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.112054 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.114472 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.114481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.114505 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.114966 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.115348 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.115355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.115360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.115588 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.116061 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.116072 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.116569 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.116852 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.117335 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:51.120299 6684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:51.120580 6684 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:51.121150 6684 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 14:47:51.122848 6493 ts_manager.cc:194] Registered new tserver with Master: 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:51.124507 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:47:51.140938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:51.266934 6689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:51.267313 6689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:51.267398 6689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:51.271927 6689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:51.272033 6689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:51.272140 6689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:51.277966 6689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:51.279376 6689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:51.280696 6689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:51.292282 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.292480 6697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.292351 6695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:51.292375 6689 server_base.cc:1034] running on GCE node
I20260929 14:47:51.293128 6689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:51.293879 6689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:51.295099 6689 hybrid_clock.cc:648] HybridClock initialized: now 1790693271295081 us; error 43 us; skew 500 ppm
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:51.303619 6689 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:47:51.304956 6689 webserver.cc:460] Webserver started at http://127.0.98.2:38945/ using document root <none> and password file <none>
I20260929 14:47:51.305703 6689 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:51.305776 6689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:51.306032 6689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:51.308753 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "36f8a8f00b9d463698757b59bc4cb205"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "1b1f23d0bf616b9aa09306f391eb1307"
server_key_iv: "9bfe37478ce4de0c4e8f74315c6623d8"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.309365 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36f8a8f00b9d463698757b59bc4cb205"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "1b1f23d0bf616b9aa09306f391eb1307"
server_key_iv: "9bfe37478ce4de0c4e8f74315c6623d8"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.313799 6689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:51.316874 6704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.318652 6689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:51.318792 6689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "36f8a8f00b9d463698757b59bc4cb205"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "1b1f23d0bf616b9aa09306f391eb1307"
server_key_iv: "9bfe37478ce4de0c4e8f74315c6623d8"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.318933 6689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:51.330466 6689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:51.334064 6689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:51.334303 6689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:51.335024 6689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:51.336078 6689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:51.336155 6689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.336246 6689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:51.336295 6689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.347530 6689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40197
I20260929 14:47:51.347554 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40197 every 8 connection(s)
I20260929 14:47:51.348737 6689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:51.359064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6689
I20260929 14:47:51.359236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:51.359596 392 external_mini_cluster.cc:1428] Setting key 313509fa954b41b08ab92cd9bbc1392d
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:51.365938 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.351178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37743 (local address 127.0.98.62:40107)
0929 14:47:51.351374 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.351381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.352439 (+ 1058us) server_negotiation.cc:407] Connection header received
0929 14:47:51.354075 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.354078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.354133 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.354246 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.355691 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.356346 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.357632 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.357971 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.358085 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.361429 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.361434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.361439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.361470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.363206 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.363838 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.363843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.363848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.363962 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.364437 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.364444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.364449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.364735 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.364872 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.365688 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.365814 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:47:51.367444 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.351616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:37743)
0929 14:47:51.352250 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.352309 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.353784 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.354450 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.354458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.354911 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.355507 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.355522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.356498 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.356500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.357494 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.357503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.358139 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.358142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.358300 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.359006 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.359030 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.361220 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.363378 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.363389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.363413 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.363722 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.364096 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.364099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.364100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.364328 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.364891 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.364901 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.365337 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.365584 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.366025 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":179,"threads_started":1}
I20260929 14:47:51.369019 6818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:51.369390 6818 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:51.370169 6818 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:47:51.371459 6493 ts_manager.cc:194] Registered new tserver with Master: 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2:40197)
I20260929 14:47:51.372293 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:47:51.390831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:51.518581 6823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:51.518957 6823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:51.519053 6823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:51.523590 6823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:51.523684 6823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:51.523772 6823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:51.529713 6823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:51.531175 6823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:51.532480 6823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:51.543948 6829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.543958 6828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:51.544027 6823 server_base.cc:1034] running on GCE node
W20260929 14:47:51.544054 6831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:51.544880 6823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:51.545619 6823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:51.546855 6823 hybrid_clock.cc:648] HybridClock initialized: now 1790693271546815 us; error 53 us; skew 500 ppm
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:51.550168 6823 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:47:51.551450 6823 webserver.cc:460] Webserver started at http://127.0.98.3:39917/ using document root <none> and password file <none>
I20260929 14:47:51.552224 6823 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:51.552313 6823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:51.552587 6823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:51.555078 6823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "ede9d1b95d9dd6f0f25520c395d2840a"
server_key_iv: "ee7e49f1dd73eba0bed2d097ec63f116"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.555764 6823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "ede9d1b95d9dd6f0f25520c395d2840a"
server_key_iv: "ee7e49f1dd73eba0bed2d097ec63f116"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.560433 6823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:47:51.563380 6838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.565037 6823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:51.565220 6823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "ede9d1b95d9dd6f0f25520c395d2840a"
server_key_iv: "ee7e49f1dd73eba0bed2d097ec63f116"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.565356 6823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:51.607411 6823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:51.611734 6823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:51.611922 6823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:51.612541 6823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:51.613436 6823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:51.613492 6823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.613533 6823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:51.613548 6823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.624397 6823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43855
I20260929 14:47:51.624411 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43855 every 8 connection(s)
I20260929 14:47:51.625355 6823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:51.629207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6823
I20260929 14:47:51.629386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:51.629809 392 external_mini_cluster.cc:1428] Setting key c7c3fb9377b7fcdad87f0ae9bff8ae20
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:51.641482 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.627605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36949 (local address 127.0.98.62:40107)
0929 14:47:51.627779 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.627784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.628749 (+ 965us) server_negotiation.cc:407] Connection header received
0929 14:47:51.630869 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.630875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.630954 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.631080 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.632754 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.633303 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.634510 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.634736 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.634855 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.637780 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.637783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.637786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.637819 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.639296 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.639832 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.639835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.639837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.639911 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.640367 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.640370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.640372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.640602 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.640723 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.641221 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.641336 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:47:51.642438 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.628011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:36949)
0929 14:47:51.628559 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.628612 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.630481 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.631325 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.631334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.631843 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.632590 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.632608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.633438 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.633442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.634374 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.634385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.634895 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.634900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.635074 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.635734 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.635751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.637561 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.639432 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.639437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.639447 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.639719 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.639999 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.640001 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.640002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.640139 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.640739 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.640744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.640986 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.641158 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.641588 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":175,"threads_started":1}
I20260929 14:47:51.644270 6952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:51.644567 6952 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:51.645143 6952 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
I20260929 14:47:51.646289 6492 ts_manager.cc:194] Registered new tserver with Master: b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855)
I20260929 14:47:51.646955 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36949
I20260929 14:47:51.656834 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 14:47:51.687314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42629
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40107,127.0.98.61:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:51.807685 6957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:51.807976 6957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:51.808019 6957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:51.812089 6957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:51.812157 6957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:51.812269 6957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:51.812314 6957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:51.812346 6957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:51.819026 6957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40107,127.0.98.61:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
W20260929 14:47:51.819244 6957 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40107,127.0.98.61:42629'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 14:47:51.820134 6957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:51.821226 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:51.830395 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.830384 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:51.830384 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:51.830790 6957 server_base.cc:1034] running on GCE node
I20260929 14:47:51.831453 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:51.832640 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:51.833829 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790693271833789 us; error 59 us; skew 500 ppm
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:51.836975 6957 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:47:51.838001 6957 webserver.cc:460] Webserver started at http://127.0.98.61:34615/ using document root <none> and password file <none>
I20260929 14:47:51.838608 6957 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:51.838675 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:51.838894 6957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:51.841286 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/instance:
uuid: "56238d34d13743fb89b379b18e888f72"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "e26cc1da03383226eeb50bc83939a122"
server_key_iv: "af7824d0213fcaa7c6036cf3ba51c9e4"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.841887 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal/instance:
uuid: "56238d34d13743fb89b379b18e888f72"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "e26cc1da03383226eeb50bc83939a122"
server_key_iv: "af7824d0213fcaa7c6036cf3ba51c9e4"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.845893 6957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:51.848711 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:51.850248 6957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:47:51.850575 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "56238d34d13743fb89b379b18e888f72"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "e26cc1da03383226eeb50bc83939a122"
server_key_iv: "af7824d0213fcaa7c6036cf3ba51c9e4"
server_key_version: "encryptionkey@0"
I20260929 14:47:51.850705 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:51.890780 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:51.894155 6957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:51.894372 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:51.901969 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42629
I20260929 14:47:51.902341 7024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42629 every 8 connection(s)
I20260929 14:47:51.903012 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:47:51.905642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 6957
I20260929 14:47:51.907207 7025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:51.911775 7025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:51.923082 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.907552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.98.61:42629)
0929 14:47:51.908138 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.908157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.908252 (+ 95us) server_negotiation.cc:407] Connection header received
0929 14:47:51.908973 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.908990 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.909354 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.909698 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.910486 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.911523 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.912461 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.913098 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.913282 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.916811 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.916816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.916826 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.916854 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.919051 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.919629 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.919637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.919645 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.919816 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.920115 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.920118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.920119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.920699 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.920936 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.921369 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.921776 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":185,"threads_started":1}
I20260929 14:47:51.924930 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.913912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.0.98.62:40107)
0929 14:47:51.914118 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.914124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.914478 (+ 354us) server_negotiation.cc:407] Connection header received
0929 14:47:51.914655 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.914658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.914726 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.914818 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.915531 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.916129 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.917467 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.917831 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.917968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.920871 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.920874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.920877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.920905 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.922637 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.923170 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.923177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.923180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.923352 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.923720 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.923723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.923724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.924038 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.924163 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.924599 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.924751 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:47:51.924947 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.913840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58172)
0929 14:47:51.914334 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.914378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.914547 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.914873 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.914879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.915278 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.915408 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.915414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.916327 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.916335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.917310 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.917326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.917982 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.917986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.918134 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.918686 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.918698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.920708 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.922775 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.922783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.922788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.923047 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.923480 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.923484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.923486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.923590 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.924203 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.924210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.924348 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.924540 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.924755 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":193,"threads_started":1}
I20260929 14:47:51.926460 7025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693271, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:47:51.935974 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.927309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60796)
0929 14:47:51.927507 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.927528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.927631 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.927854 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.927857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.928052 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.928145 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.928151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.929334 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.929341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.929886 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.929894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.930352 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.930356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.930447 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.931049 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.931061 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.932520 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.934196 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.934199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.934201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.934418 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.934806 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.934809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.934811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.934900 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.935358 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.935360 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.935470 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.935576 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.935720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 14:47:51.935999 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.927448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.98.61:42629)
0929 14:47:51.927579 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.927585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.927600 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:47:51.927708 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.927714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.927759 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.927845 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.928280 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.929220 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.930001 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.930242 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.930352 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.932695 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.932698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.932701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.932722 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.934072 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.934537 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.934541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.934542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.934625 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.935034 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.935037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.935038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.935244 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.935330 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.935665 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.935774 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:47:51.939602 7025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Bootstrap starting.
I20260929 14:47:51.942390 7025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:51.943260 7025 log.cc:824] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:51.945607 7025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: No bootstrap required, opened a new log
I20260929 14:47:51.949343 7025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } }
I20260929 14:47:51.949640 7025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:51.949731 7025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56238d34d13743fb89b379b18e888f72, State: Initialized, Role: FOLLOWER
I20260929 14:47:51.950312 7025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } }
I20260929 14:47:51.950939 7031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } } }
I20260929 14:47:51.951167 7031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:51.951864 7025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:47:51.957396 7042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:51.957499 7042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:51.960119 6957 master_runner.cc:428] Detected that this master 56238d34d13743fb89b379b18e888f72 is joining an existing cluster
I20260929 14:47:51.960242 6957 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42629
I20260929 14:47:51.972713 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.963327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.98.61:42629)
0929 14:47:51.963527 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.963530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.963541 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:47:51.963681 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.963684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.963724 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.963814 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.964490 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.965209 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.966078 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.966454 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.966581 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.968679 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.968683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.968685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.968713 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.970736 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971313 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.971319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.971321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.971426 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971753 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.971757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.971759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.971959 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.972073 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.972360 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.972528 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":142}
I20260929 14:47:51.972728 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.962720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60810)
0929 14:47:51.963414 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.963429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.963564 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.963875 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.963878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.964165 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.964356 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.964363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.965312 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.965317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.965964 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.965972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.967016 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.967019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.967216 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.967826 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.967846 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.968551 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.970856 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.970859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.970862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.971218 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.971533 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.971538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.971638 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.972059 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.972062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.972147 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.972310 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.972476 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":135,"thread_start_us":297,"threads_started":1}
I20260929 14:47:51.973044 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.962701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.62:40107)
0929 14:47:51.963154 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.963159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.963176 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:47:51.963228 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.963232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.963288 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.963369 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.964143 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.965011 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.966565 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.966831 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.966912 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.968783 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.968786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.968789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.968813 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.970850 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971455 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.971461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.971466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.971549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971885 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.971890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.971892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.972226 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.972338 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.972718 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.972866 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20260929 14:47:51.973855 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.962644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58186)
0929 14:47:51.963014 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.963033 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.963142 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.963583 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.963585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.963898 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.964023 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.964029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.965146 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.965150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.966421 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.966433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.966986 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.966993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.967227 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.968009 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.968027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.968627 (+ 600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.970957 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.970960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.970962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.971344 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.971657 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.971660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.971662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.971753 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.972385 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.972388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.972517 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.973496 (+ 979us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.973663 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":15,"thread_start_us":145,"threads_started":1}
Sep 29 14:47:51 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 14:47:51.988941 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.979580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.98.61:42629)
0929 14:47:51.979704 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:51.979710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:51.980112 (+ 402us) server_negotiation.cc:407] Connection header received
0929 14:47:51.980396 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:51.980399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:51.980457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:51.980552 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:51.981211 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.981728 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.982632 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.982960 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.983082 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.985470 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:51.985475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:51.985486 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:51.985507 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:51.986946 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.987463 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.987470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.987472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.987583 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:51.987868 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:51.987871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:51.987873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:51.988093 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:51.988225 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:51.988642 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 14:47:51.988754 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:47:51.988943 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:51.979579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60818)
0929 14:47:51.979977 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:51.980039 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:51.980236 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:51.980585 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:51.980591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:51.980972 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:51.981095 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.981101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.981836 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.981841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.982516 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:51.982524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:51.983081 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:51.983084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:51.983205 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:51.983741 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:51.983752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:51.985331 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:51.987106 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.987116 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.987123 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.987361 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.987678 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:51.987681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:51.987683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:51.987770 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:51.988226 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:51.988234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:51.988366 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 14:47:51.988532 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 14:47:51.988755 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":138,"threads_started":1}
I20260929 14:47:51.990549 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42629: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "56238d34d13743fb89b379b18e888f72"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42629
}
attrs {
promote: true
}
}
dest_uuid: "e2c48ccb36f94260a3b4ed461872d21e"
cas_config_opid_index: -1
I20260929 14:47:51.991313 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 14:47:51.993968 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 56238d34d13743fb89b379b18e888f72 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } }
I20260929 14:47:51.994837 6999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:51.996025 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42629
I20260929 14:47:51.996357 6999 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 14:47:51.996337 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } }
I20260929 14:47:51.996428 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:51.996605 6957 master.cc:537] Master@127.0.98.61:42629 shutting down...
I20260929 14:47:51.997151 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:47:51.999374 6477 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 14:47:52.001564 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e -> Peer 56238d34d13743fb89b379b18e888f72 (127.0.98.61:42629): Couldn't send request to peer 56238d34d13743fb89b379b18e888f72. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:47:52.002523 6957 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:52.002867 6957 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:52.003014 6957 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 14:47:52.014955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35353
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:47:52.018635 6957 master.cc:552] Master@127.0.98.61:42629 shutdown complete.
I20260929 14:47:52.018718 6957 master_runner.cc:305] Clearing existing system tablet
I20260929 14:47:52.018978 6957 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.019038 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.020521 6957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:52.021435 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.021616 6957 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.021739 6957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:52.021772 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "56238d34d13743fb89b379b18e888f72"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "e26cc1da03383226eeb50bc83939a122"
server_key_iv: "af7824d0213fcaa7c6036cf3ba51c9e4"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.024641 6957 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:52.027100 6957 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:47:52.027201 6957 log.cc:1196] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 14:47:52.027464 6957 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Deleting consensus metadata
I20260929 14:47:52.027637 6957 master_runner.cc:315] Copying system tablet from 127.0.98.62:40107
I20260929 14:47:52.028895 6957 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40107
I20260929 14:47:52.038975 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.029499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58190)
0929 14:47:52.029815 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.029831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.029935 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.030318 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.030321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.030536 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.030675 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.030682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.031886 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.031889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.032878 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.032890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.033600 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.033603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.033717 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.034372 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.034386 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.034819 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.037196 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.037199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.037202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.037483 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.037818 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.037821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.037823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.037915 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.038391 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.038394 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.038500 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.038614 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.038767 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260929 14:47:52.039402 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.029782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.98.62:40107)
0929 14:47:52.029943 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.029949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.030029 (+ 80us) server_negotiation.cc:407] Connection header received
0929 14:47:52.030100 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.030104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.030167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.030269 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.030835 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.031745 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.033028 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.033458 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.033593 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.034985 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.034989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.034993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.035021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.037061 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.037598 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.037603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.037607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.037708 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.038021 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.038024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.038026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.038282 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.038388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.038691 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.038826 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:47:52.040004 6513 tablet_copy_service.cc:140] P e2c48ccb36f94260a3b4ed461872d21e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 56238d34d13743fb89b379b18e888f72 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58190)
I20260929 14:47:52.040238 6513 tablet_copy_service.cc:161] P e2c48ccb36f94260a3b4ed461872d21e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 56238d34d13743fb89b379b18e888f72 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58190: session id = 56238d34d13743fb89b379b18e888f72-00000000000000000000000000000000
I20260929 14:47:52.042299 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:47:52.044399 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:47:52.046819 6957 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: tablet copy: Starting download of 0 data blocks...
I20260929 14:47:52.047019 6957 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: tablet copy: Starting download of 1 WAL segments...
I20260929 14:47:52.049089 6957 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:47:52.050999 6513 tablet_copy_service.cc:342] P e2c48ccb36f94260a3b4ed461872d21e: Request end of tablet copy session 56238d34d13743fb89b379b18e888f72-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58190
I20260929 14:47:52.051143 6513 tablet_copy_service.cc:433] P e2c48ccb36f94260a3b4ed461872d21e: ending tablet copy session 56238d34d13743fb89b379b18e888f72-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 56238d34d13743fb89b379b18e888f72
I20260929 14:47:52.060221 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:52.062130 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.062685 6957 server_base.cc:1034] running on GCE node
W20260929 14:47:52.062631 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.062948 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.063261 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:52.063618 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:52.064774 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790693272064742 us; error 52 us; skew 500 ppm
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:52.066604 6957 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:47:52.067106 6957 webserver.cc:460] Webserver started at http://127.0.98.61:38841/ using document root <none> and password file <none>
I20260929 14:47:52.067315 6957 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.067366 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.069128 6957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 14:47:52.070359 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.070696 6957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:52.070775 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "56238d34d13743fb89b379b18e888f72"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "e26cc1da03383226eeb50bc83939a122"
server_key_iv: "af7824d0213fcaa7c6036cf3ba51c9e4"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.070869 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.104064 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:52.106745 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:52.112285 6957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42629
I20260929 14:47:52.112417 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42629 every 8 connection(s)
I20260929 14:47:52.112977 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:47:52.115034 7131 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:47:52.116211 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Bootstrap starting.
I20260929 14:47:52.127136 6684 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:52.127472 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:47:52.127887 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: Bootstrap complete.
I20260929 14:47:52.128659 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } }
I20260929 14:47:52.128971 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 56238d34d13743fb89b379b18e888f72, State: Initialized, Role: LEARNER
I20260929 14:47:52.129134 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } }
I20260929 14:47:52.129671 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:52.130250 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } }
I20260929 14:47:52.130424 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:47:52.134492 7145 catalog_manager.cc:1259] Loaded cluster ID: 2f615b46235e411b87ca818eb10de484
I20260929 14:47:52.134583 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: loading cluster ID for follower catalog manager: success
I20260929 14:47:52.137321 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: acquiring CA information for follower catalog manager: success
I20260929 14:47:52.139025 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 14:47:52.174340 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:52.174683 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:52.174729 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:52.179015 7059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:52.179092 7059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:52.179121 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:52.179139 7059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:52.179155 7059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:52.185025 7059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:52.186386 7059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:52.187583 7059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:52.197274 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.197289 7150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.197575 7059 server_base.cc:1034] running on GCE node
W20260929 14:47:52.197289 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.198446 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:52.199869 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:52.201071 7059 hybrid_clock.cc:648] HybridClock initialized: now 1790693272201032 us; error 56 us; skew 500 ppm
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:52.204373 7059 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:47:52.205683 7059 webserver.cc:460] Webserver started at http://127.0.98.60:35425/ using document root <none> and password file <none>
I20260929 14:47:52.206388 7059 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.206494 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.206777 7059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:52.209435 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/instance:
uuid: "b899fad8922247849d3853c95394b98c"
format_stamp: "Formatted at 2026-09-29 14:47:52 on dist-test-slave-5hd1"
server_key: "108f31b22ab4e33c185828bb9b3f35a5"
server_key_iv: "9d07ed9372e3b652dd2771d24447a219"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.210139 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal/instance:
uuid: "b899fad8922247849d3853c95394b98c"
format_stamp: "Formatted at 2026-09-29 14:47:52 on dist-test-slave-5hd1"
server_key: "108f31b22ab4e33c185828bb9b3f35a5"
server_key_iv: "9d07ed9372e3b652dd2771d24447a219"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.214852 7059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:47:52.217960 7160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.219694 7059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:52.219880 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "b899fad8922247849d3853c95394b98c"
format_stamp: "Formatted at 2026-09-29 14:47:52 on dist-test-slave-5hd1"
server_key: "108f31b22ab4e33c185828bb9b3f35a5"
server_key_iv: "9d07ed9372e3b652dd2771d24447a219"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.220026 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.268594 7059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:52.273180 7059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:52.273442 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:52.282616 7059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35353
I20260929 14:47:52.282965 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35353 every 8 connection(s)
I20260929 14:47:52.283902 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:47:52.286234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7059
I20260929 14:47:52.288460 7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:52.293943 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:47:52.302850 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.287982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.98.60:35353)
0929 14:47:52.288533 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.288551 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.288626 (+ 75us) server_negotiation.cc:407] Connection header received
0929 14:47:52.289375 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.289395 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.289765 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.290205 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.291190 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.292376 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.293602 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.294103 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.294250 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.297714 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.297719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.297728 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.297760 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.299804 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.300328 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.300333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.300337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.300443 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.300724 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.300729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.300731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.301152 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.301383 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.301612 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.301944 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":240,"threads_started":1}
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:52.307341 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.296612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.98.62:40107)
0929 14:47:52.296794 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.296799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.297086 (+ 287us) server_negotiation.cc:407] Connection header received
0929 14:47:52.297311 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.297314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.297374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.297464 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.298370 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.299255 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.300254 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.300503 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.300621 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.303824 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.303828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.303830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.303857 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.305696 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.306120 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.306123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.306125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.306200 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.306500 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.306503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.306504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.306700 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.306783 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.307102 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.307206 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:47:52.307348 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.296463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58194)
0929 14:47:52.296930 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.297003 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.297190 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.297518 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.297524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.297979 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.298182 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.298192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.299387 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.299390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.300088 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.300096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.300631 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.300634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.300756 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.302058 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.302070 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.303669 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.305808 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.305814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.305818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.306028 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.306305 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.306308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.306309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.306403 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.306810 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.306815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.306916 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.307040 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.307191 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260929 14:47:52.308817 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:47:52.317291 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.309406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60820)
0929 14:47:52.309524 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.309540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.309610 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.310042 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.310044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.310179 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.310266 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.310271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.311196 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.311199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.311744 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.311751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.312214 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.312220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.312299 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.312735 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.312746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.314026 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.315851 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.315853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.315855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.316043 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.316367 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.316369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.316370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.316443 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.316866 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.316871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.316975 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.317074 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.317172 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 14:47:52.317437 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.309521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.98.61:42629)
0929 14:47:52.309817 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.309821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.309834 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:52.309879 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.309881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.309927 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.310014 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.310377 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.311093 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.311854 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.312077 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.312157 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.314167 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.314170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.314172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.314194 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.315736 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.316138 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.316141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.316142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.316278 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.316532 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.316535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.316536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.316723 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.316803 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.317145 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.317258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":132,"threads_started":1}
I20260929 14:47:52.318286 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:47:52.327685 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.318797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:33964)
0929 14:47:52.318929 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.318941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.319008 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.319247 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.319249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.319387 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.319467 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.319472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.320829 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.320832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.321408 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.321416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.322155 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.322158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.322272 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.322819 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.322838 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.324595 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.326393 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.326395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.326397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.326597 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.326857 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.326859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.326861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.326933 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.327278 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.327280 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.327359 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.327457 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.327561 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:47:52.327895 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.318922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.0.98.60:35353)
0929 14:47:52.319060 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.319064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.319077 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:52.319108 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.319110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.319142 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.319217 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.319573 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.320695 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.321525 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.322008 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.322128 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.324796 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.324799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.324801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.324819 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.326274 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.326698 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.326701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.326702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.326773 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.327016 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.327018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.327019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.327193 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.327270 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.327499 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.327645 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":58}
I20260929 14:47:52.331331 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Bootstrap starting.
I20260929 14:47:52.334203 7213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:52.335055 7213 log.cc:824] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:52.337541 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: No bootstrap required, opened a new log
I20260929 14:47:52.341094 7213 raft_consensus.cc:348] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } }
I20260929 14:47:52.341372 7213 raft_consensus.cc:374] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:52.341462 7213 raft_consensus.cc:729] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b899fad8922247849d3853c95394b98c, State: Initialized, Role: FOLLOWER
I20260929 14:47:52.341964 7213 consensus_queue.cc:260] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } }
I20260929 14:47:52.342541 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } } }
I20260929 14:47:52.342724 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:52.343392 7213 sys_catalog.cc:564] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:47:52.349339 7230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:47:52.349449 7230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:47:52.353130 7059 master_runner.cc:428] Detected that this master b899fad8922247849d3853c95394b98c is joining an existing cluster
I20260929 14:47:52.353214 7059 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35353
I20260929 14:47:52.357618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6555
I20260929 14:47:52.366577 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.356035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60824 (local address 127.0.98.61:42629)
0929 14:47:52.356221 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.356226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.356901 (+ 675us) server_negotiation.cc:407] Connection header received
0929 14:47:52.356990 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.356995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.357053 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.357127 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.358297 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.359193 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.360666 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.361108 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.361200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.362765 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.362772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.362778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.362809 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.364564 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.365047 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.365050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.365052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.365134 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.365452 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.365458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.365463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.365701 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.365792 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.366178 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.366323 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:47:52.367254 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.355602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60824)
0929 14:47:52.356791 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.356804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.356904 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.357249 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.357255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.357592 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.358151 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.358164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.359341 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.359347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.360525 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.360540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.361370 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.361376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.361515 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.362010 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.362030 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.362606 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.364684 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.364687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.364689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.364936 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.365249 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.365255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.365259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.365353 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.365867 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.365873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.365995 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.366802 (+ 807us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.366974 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260929 14:47:52.369012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42085
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40107
--webserver_port=36679
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:47:52.371364 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.356335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:33970)
0929 14:47:52.362043 (+ 5708us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.362058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.362191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.362480 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.362483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.362745 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.362866 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.362872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.364477 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.364480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.365387 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.365396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.366055 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.366058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.366159 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.366683 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.366698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.367164 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.369694 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.369700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.369705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.370042 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.370368 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.370373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.370378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.370499 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.370949 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.370951 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.371036 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.371125 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.371241 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5631,"thread_start_us":99,"threads_started":1}
I20260929 14:47:52.371537 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.357196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.98.60:35353)
0929 14:47:52.357391 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.357396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.362134 (+ 4738us) server_negotiation.cc:407] Connection header received
0929 14:47:52.362277 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.362280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.362319 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.362363 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.363011 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.364339 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.365508 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.365841 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.365907 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.367300 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.367304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.367307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.367326 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.369580 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.370163 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.370166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.370168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.370256 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.370610 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.370613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.370615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.370847 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.370953 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.371204 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.371336 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 14:47:52.374830 6818 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:52.384434 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.356701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.98.62:40107)
0929 14:47:52.356825 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.356829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.356843 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:52.356893 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.356896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.356947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.357043 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.357750 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.358548 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.359885 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.360261 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.360345 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.380862 (+ 20517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.380865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.380868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.380895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.382629 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.383133 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.383138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.383140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.383218 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.383526 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.383531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.383533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.383732 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.383847 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.384116 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.384275 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:47:52.384450 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.355372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58202)
0929 14:47:52.356408 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.356427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.356542 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.357062 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.357065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.357407 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.357565 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.357574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.358713 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.358717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.359745 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.359755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.360544 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.360548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.360712 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.361407 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.361422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.380651 (+ 19229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.382779 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.382782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.382785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.383024 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.383323 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.383327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.383329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.383432 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.383833 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.383836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.383955 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.384066 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.384253 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":45,"thread_start_us":121,"threads_started":1}
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693270, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:47:52.395787 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.388032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:33976)
0929 14:47:52.388311 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.388322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.388420 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.388693 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.388696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.388885 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.388964 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.388969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.389685 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.389690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.390228 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.390243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.390687 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.390691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.390782 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.391230 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.391254 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.392553 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.394340 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.394345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.394347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.394533 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.394815 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.394820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.394822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.394901 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.395272 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.395275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.395366 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.395472 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.395620 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 14:47:52.395787 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.388073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.60:35353)
0929 14:47:52.388201 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.388206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.388380 (+ 174us) server_negotiation.cc:407] Connection header received
0929 14:47:52.388537 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.388539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.388593 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.388680 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.389074 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.389577 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.390353 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.390581 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.390689 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.392684 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.392687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.392690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.392712 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.394225 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.394631 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.394637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.394638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.394717 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.394999 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.395002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.395004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.395169 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.395278 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.395528 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.395629 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:47:52.397102 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35353: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b899fad8922247849d3853c95394b98c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35353
}
attrs {
promote: true
}
}
dest_uuid: "e2c48ccb36f94260a3b4ed461872d21e"
cas_config_opid_index: 5
I20260929 14:47:52.397688 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } }
I20260929 14:47:52.398927 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b899fad8922247849d3853c95394b98c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.399344 7187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:52.400542 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35353
I20260929 14:47:52.400712 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.400818 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:52.401232 7187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 14:47:52.401703 7059 master.cc:537] Master@127.0.98.60:35353 shutting down...
I20260929 14:47:52.402226 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:47:52.403990 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e -> Peer b899fad8922247849d3853c95394b98c (127.0.98.60:35353): Couldn't send request to peer b899fad8922247849d3853c95394b98c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:47:52.406255 7219 raft_consensus.cc:482] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:52.406423 7219 raft_consensus.cc:504] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } }
I20260929 14:47:52.407356 7219 leader_election.cc:288] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107), 56238d34d13743fb89b379b18e888f72 (127.0.98.61:42629)
I20260929 14:47:52.407680 6503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b899fad8922247849d3853c95394b98c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c48ccb36f94260a3b4ed461872d21e" is_pre_election: true
I20260929 14:47:52.407841 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b899fad8922247849d3853c95394b98c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56238d34d13743fb89b379b18e888f72" is_pre_election: true
I20260929 14:47:52.407999 7097 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Handling vote request from an unknown peer b899fad8922247849d3853c95394b98c
I20260929 14:47:52.408144 7097 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate b899fad8922247849d3853c95394b98c for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 14:47:52.408564 7161 leader_election.cc:302] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b899fad8922247849d3853c95394b98c; no voters: 56238d34d13743fb89b379b18e888f72, e2c48ccb36f94260a3b4ed461872d21e
I20260929 14:47:52.408821 7219 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:47:52.411827 7059 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:52.412147 7059 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:52.412355 7059 tablet_replica.cc:330] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: stopping tablet replica
I20260929 14:47:52.415788 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.400316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.61:42629)
0929 14:47:52.400459 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.400463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.404513 (+ 4050us) server_negotiation.cc:407] Connection header received
0929 14:47:52.410753 (+ 6240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.410757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.410827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.410939 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:52.411367 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.412149 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.413623 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.414218 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.414320 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.414462 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.415550 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.415638 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:47:52.416224 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.398759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60834)
0929 14:47:52.404384 (+ 5625us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.404399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.404535 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.410960 (+ 6425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.410963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.410981 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:52.411201 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.411214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.412332 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.412335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.413482 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.413493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.415263 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.415270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.415370 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.415422 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.415978 (+ 556us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.416059 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5539}
I20260929 14:47:52.416749 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 14:47:52.417346 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:47:52.418817 7059 master.cc:552] Master@127.0.98.60:35353 shutdown complete.
I20260929 14:47:52.418869 7059 master_runner.cc:305] Clearing existing system tablet
I20260929 14:47:52.419106 7059 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.419152 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.419708 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b899fad8922247849d3853c95394b98c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.420502 7059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 14:47:52.421294 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.421478 7059 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.421588 7059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:52.421620 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "b899fad8922247849d3853c95394b98c"
format_stamp: "Formatted at 2026-09-29 14:47:52 on dist-test-slave-5hd1"
server_key: "108f31b22ab4e33c185828bb9b3f35a5"
server_key_iv: "9d07ed9372e3b652dd2771d24447a219"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.421824 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c48ccb36f94260a3b4ed461872d21e. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.421921 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:47:52.421993 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: true } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.422053 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: LEARNER
I20260929 14:47:52.423930 7059 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:52.424042 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: attempting to promote NON_VOTER 56238d34d13743fb89b379b18e888f72 to VOTER
I20260929 14:47:52.424554 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } }
I20260929 14:47:52.425927 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.425663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:33992)
0929 14:47:52.425772 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.425844 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 14:47:52.426358 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 LEARNER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 14:47:52.426705 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.426455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:34002)
0929 14:47:52.426583 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.426639 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 14:47:52.426818 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e -> Peer b899fad8922247849d3853c95394b98c (127.0.98.60:35353): Couldn't send request to peer b899fad8922247849d3853c95394b98c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:47:52.426832 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:47:52.428318 7059 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:47:52.428409 7059 log.cc:1196] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 14:47:52.428680 7059 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Deleting consensus metadata
I20260929 14:47:52.428639 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 56238d34d13743fb89b379b18e888f72 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.428838 7059 master_runner.cc:315] Copying system tablet from 127.0.98.62:40107
I20260929 14:47:52.430117 7059 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40107
I20260929 14:47:52.430099 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.430195 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:52.430325 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.430395 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:52.431119 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 56238d34d13743fb89b379b18e888f72 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.432804 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.432924 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:52.443231 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.430961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.98.62:40107)
0929 14:47:52.431058 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.431062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.436547 (+ 5485us) server_negotiation.cc:407] Connection header received
0929 14:47:52.436714 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.436717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.436764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.436857 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.437317 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.437839 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.438648 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.438879 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.438989 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.440105 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.440109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.440112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.440133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.441622 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.442038 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.442043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.442045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.442125 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.442397 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.442402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.442404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.442579 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.442686 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.442961 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.443062 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:47:52.443581 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.431154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58206)
0929 14:47:52.436447 (+ 5293us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.436460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.436597 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.436873 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.436876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.437091 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.437184 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.437189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.437944 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.437950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.438537 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.438544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.438993 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.438996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.439074 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.439634 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.439649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.439965 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.441733 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.441739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.441741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.441942 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.442220 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.442223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.442225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.442309 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.442676 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.442679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.442777 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.442885 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.443052 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5213,"thread_start_us":82,"threads_started":1}
I20260929 14:47:52.444039 6513 tablet_copy_service.cc:140] P e2c48ccb36f94260a3b4ed461872d21e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b899fad8922247849d3853c95394b98c ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58206)
I20260929 14:47:52.444159 6513 tablet_copy_service.cc:161] P e2c48ccb36f94260a3b4ed461872d21e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b899fad8922247849d3853c95394b98c at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58206: session id = b899fad8922247849d3853c95394b98c-00000000000000000000000000000000
I20260929 14:47:52.445317 6513 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:47:52.446548 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:47:52.448604 7059 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: tablet copy: Starting download of 0 data blocks...
I20260929 14:47:52.448822 7059 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: tablet copy: Starting download of 1 WAL segments...
I20260929 14:47:52.450290 7059 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:47:52.451900 6513 tablet_copy_service.cc:342] P e2c48ccb36f94260a3b4ed461872d21e: Request end of tablet copy session b899fad8922247849d3853c95394b98c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:58206
I20260929 14:47:52.451996 6513 tablet_copy_service.cc:433] P e2c48ccb36f94260a3b4ed461872d21e: ending tablet copy session b899fad8922247849d3853c95394b98c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b899fad8922247849d3853c95394b98c
I20260929 14:47:52.461287 7059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:52.463142 7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.463150 7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.464408 7059 server_base.cc:1034] running on GCE node
W20260929 14:47:52.467839 7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.468112 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:52.468531 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:52.469714 7059 hybrid_clock.cc:648] HybridClock initialized: now 1790693272469714 us; error 57 us; skew 500 ppm
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:52.471585 7059 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:47:52.472206 7059 webserver.cc:460] Webserver started at http://127.0.98.60:32773/ using document root <none> and password file <none>
I20260929 14:47:52.472409 7059 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.472466 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.473675 7059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:52.474679 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.474915 7059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:52.475005 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "b899fad8922247849d3853c95394b98c"
format_stamp: "Formatted at 2026-09-29 14:47:52 on dist-test-slave-5hd1"
server_key: "108f31b22ab4e33c185828bb9b3f35a5"
server_key_iv: "9d07ed9372e3b652dd2771d24447a219"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.475111 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.485056 7059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:52.487061 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:52.492010 7059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35353
I20260929 14:47:52.492642 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:47:52.494154 7313 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:47:52.494997 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Bootstrap starting.
I20260929 14:47:52.500393 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35353 every 8 connection(s)
I20260929 14:47:52.503996 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:47:52.504325 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Bootstrap complete.
I20260929 14:47:52.505026 7313 raft_consensus.cc:348] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } }
I20260929 14:47:52.505180 7313 raft_consensus.cc:729] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b899fad8922247849d3853c95394b98c, State: Initialized, Role: LEARNER
I20260929 14:47:52.505291 7313 consensus_queue.cc:260] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } }
I20260929 14:47:52.505579 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.505762 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: LEARNER
I20260929 14:47:52.505772 7313 sys_catalog.cc:564] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:52.509788 7327 catalog_manager.cc:1259] Loaded cluster ID: 2f615b46235e411b87ca818eb10de484
I20260929 14:47:52.509864 7327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: loading cluster ID for follower catalog manager: success
I20260929 14:47:52.511461 7327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: acquiring CA information for follower catalog manager: success
I20260929 14:47:52.512738 7327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 14:47:52.562251 7239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:52.562738 7239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:52.562811 7239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:52.569723 7239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:52.569859 7239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:52.569963 7239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:52.579252 7239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36679
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:52.581024 7239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:52.582423 7239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:52.594357 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.594511 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.594588 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.594856 7239 server_base.cc:1034] running on GCE node
I20260929 14:47:52.595402 7239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:52.596158 7239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:52.597401 7239 hybrid_clock.cc:648] HybridClock initialized: now 1790693272597379 us; error 41 us; skew 500 ppm
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:52.600423 7239 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:47:52.601428 7239 webserver.cc:460] Webserver started at http://127.0.98.1:36679/ using document root <none> and password file <none>
I20260929 14:47:52.602021 7239 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.602095 7239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.607275 7239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:47:52.609921 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.611505 7239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:52.611627 7239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "8681f175c4e549a487c8eda1ed0a934d"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "b92ee93dd73aaa917bd2d0bcc466af9d"
server_key_iv: "a5e5e78585169a7417603374b838d7c9"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.612249 7239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.641088 7239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:52.644673 7239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:52.644847 7239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:52.645406 7239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:52.646353 7239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:52.646404 7239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.646446 7239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:52.646514 7239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.649111 6952 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:52.658763 7239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42085
I20260929 14:47:52.658900 7455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42085 every 8 connection(s)
I20260929 14:47:52.660233 7239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:52.662215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7239
I20260929 14:47:52.662456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6689
I20260929 14:47:52.672130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40197
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40107
--webserver_port=38945
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:47:52.683852 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.666156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35403 (local address 127.0.98.61:42629)
0929 14:47:52.666387 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.666395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.666810 (+ 415us) server_negotiation.cc:407] Connection header received
0929 14:47:52.667441 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.667447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.667516 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.667585 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.669232 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670101 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.672440 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.672784 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.672865 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.678400 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.678405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.678409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.678445 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.680843 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.681537 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.681541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.681543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.681658 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.682126 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.682129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.682131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.682389 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.682478 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.683478 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.683656 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:47:52.683852 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.665131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33165 (local address 127.0.98.62:40107)
0929 14:47:52.665347 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.665356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.667069 (+ 1713us) server_negotiation.cc:407] Connection header received
0929 14:47:52.667332 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.667339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.667413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.667514 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.669257 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670186 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.672474 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.672905 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.673012 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.678451 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.678455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.678458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.678495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.680653 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.681537 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.681541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.681543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.681656 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.682134 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.682138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.682140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.682389 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.682478 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.683478 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.683648 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 14:47:52.684382 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.663453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37933 (local address 127.0.98.60:35353)
0929 14:47:52.666490 (+ 3037us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.666504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.666528 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:47:52.666764 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.666772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.666862 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.667029 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.669292 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670432 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.672458 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.672802 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.672870 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.678558 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.678565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.678570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.678604 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.681969 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.682615 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.682619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.682623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.682718 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.683098 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.683104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.683107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.683402 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.683501 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.683947 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.684095 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2940,"thread_start_us":138,"threads_started":1}
I20260929 14:47:52.686291 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.664678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:33165)
0929 14:47:52.666947 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.666958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.667159 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.667697 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.667703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.668131 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.669046 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.669076 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670360 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.670366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.672296 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.672312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.673237 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.673243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.673448 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.674479 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.674510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.678194 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.680842 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.680851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.680879 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.681406 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.681768 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.681772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.681774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.682009 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.682554 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.682563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.683141 (+ 578us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.684077 (+ 936us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.684769 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"mutex_wait_us":311,"spinlock_wait_cycles":28160,"thread_start_us":106,"threads_started":1}
I20260929 14:47:52.686295 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.663167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.98.1:37933)
0929 14:47:52.665211 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.665278 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.666466 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.667284 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.667295 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.668078 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.669048 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.669069 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670565 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.670568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.672294 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.672300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.673076 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.673080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.673426 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.674636 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.674654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.678385 (+ 3731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.682153 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.682159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.682164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.682478 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.682845 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.682851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.682856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.682977 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.683571 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.683576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.683726 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.685162 (+ 1436us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.685394 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"mutex_wait_us":476,"spinlock_wait_cycles":25856,"thread_start_us":852,"threads_started":1}
I20260929 14:47:52.686296 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.663167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.98.1:35403)
0929 14:47:52.666709 (+ 3542us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.666725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.667311 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.667985 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.667988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.668278 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.669048 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.669069 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.670310 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.670317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.672289 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.672300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.673076 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.673080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.673421 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.674479 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.674505 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.678097 (+ 3592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.680971 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.680977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.680981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.681406 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.681768 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.681772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.681774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.682010 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.682554 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.682563 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.683120 (+ 557us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.684270 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.684824 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3438,"mutex_wait_us":616,"spinlock_wait_cycles":26624,"thread_start_us":191,"threads_started":1}
I20260929 14:47:52.688069 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35353
I20260929 14:47:52.688159 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:52.688297 7458 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42629
I20260929 14:47:52.688422 7458 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.688418 7457 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.688418 7456 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.689126 7458 heartbeater.cc:511] Master 127.0.98.61:42629 requested a full tablet report, sending...
I20260929 14:47:52.689359 7456 heartbeater.cc:511] Master 127.0.98.60:35353 requested a full tablet report, sending...
I20260929 14:47:52.689123 7457 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
I20260929 14:47:52.691147 7277 ts_manager.cc:194] Registered new tserver with Master: 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:52.691102 7088 ts_manager.cc:194] Registered new tserver with Master: 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:52.692131 6492 ts_manager.cc:194] Re-registered known tserver with Master: 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:52.693040 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33165
W20260929 14:47:52.805361 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:52.805681 7464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:52.805749 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:52.810076 7464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:52.810148 7464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:52.810217 7464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:52.816860 7464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38945
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:52.818183 7464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:52.819339 7464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:52.830152 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:52.830242 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.830678 7464 server_base.cc:1034] running on GCE node
W20260929 14:47:52.830255 7472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:52.831174 7464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:52.831954 7464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:52.833146 7464 hybrid_clock.cc:648] HybridClock initialized: now 1790693272833129 us; error 48 us; skew 500 ppm
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:52.836243 7464 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:47:52.837225 7464 webserver.cc:460] Webserver started at http://127.0.98.2:38945/ using document root <none> and password file <none>
I20260929 14:47:52.837814 7464 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:52.837924 7464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:52.842664 7464 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 14:47:52.845423 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.847060 7464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:52.847229 7464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "36f8a8f00b9d463698757b59bc4cb205"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "1b1f23d0bf616b9aa09306f391eb1307"
server_key_iv: "9bfe37478ce4de0c4e8f74315c6623d8"
server_key_version: "encryptionkey@0"
I20260929 14:47:52.847824 7464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:52.873092 7464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:52.876557 7464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:52.876757 7464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:52.877373 7464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:52.878245 7464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:52.878315 7464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.878378 7464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:52.878420 7464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:52.889556 7464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40197
I20260929 14:47:52.889617 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40197 every 8 connection(s)
I20260929 14:47:52.890611 7464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:52.892258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7464
I20260929 14:47:52.892452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6823
I20260929 14:47:52.902590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43855
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40107
--webserver_port=39917
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:47:52 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693272, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:52.911842 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.897755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39763 (local address 127.0.98.61:42629)
0929 14:47:52.897914 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.897919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.897932 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:52.897977 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.897980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.898025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.898113 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.900242 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901139 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903054 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.903404 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903476 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.907886 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.907890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.907893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.907920 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.909595 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910209 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.910214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.910216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.910296 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910708 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.910711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.910713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.910911 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.911015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.911546 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.911673 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:47:52.912624 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.893265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37221 (local address 127.0.98.60:35353)
0929 14:47:52.893494 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.893500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.897035 (+ 3535us) server_negotiation.cc:407] Connection header received
0929 14:47:52.897191 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.897198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.897284 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.897382 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.900211 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901103 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903086 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.903511 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.907614 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.907621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.907625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.907669 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.909705 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910308 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.910314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.910318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.910427 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910930 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.910937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.910942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.911253 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.911347 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.912315 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.912479 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 14:47:52.913625 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.896556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.98.2:37221)
0929 14:47:52.896877 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.896894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.897055 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.898507 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.898510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.898979 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.899998 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.900005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901302 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.901306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.902915 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.902946 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.903807 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.903994 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.905017 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.905036 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.907332 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.909856 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.909862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.909866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.910203 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.910528 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.910538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.910785 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.911438 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.911444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.911789 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.912228 (+ 439us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.912463 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":558,"spinlock_wait_cycles":4864,"thread_start_us":121,"threads_started":1}
I20260929 14:47:52.913929 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.893533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.98.2:39763)
0929 14:47:52.894682 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.894747 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.896215 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.898178 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.898185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.898907 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.899992 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.900007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901311 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.901317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.902883 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.902896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.903754 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.903758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.903983 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.904863 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.904891 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.907688 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.909729 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.909736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.909757 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.910103 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.910404 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.910408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.910410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.910604 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.911025 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.911034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.911303 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.911509 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.911887 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":204,"spinlock_wait_cycles":8320,"thread_start_us":668,"threads_started":1}
I20260929 14:47:52.915295 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35353
I20260929 14:47:52.915360 7595 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42629
I20260929 14:47:52.915611 7595 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.915608 7593 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.916402 7593 heartbeater.cc:511] Master 127.0.98.60:35353 requested a full tablet report, sending...
I20260929 14:47:52.916404 7595 heartbeater.cc:511] Master 127.0.98.61:42629 requested a full tablet report, sending...
I20260929 14:47:52.917824 7277 ts_manager.cc:194] Registered new tserver with Master: 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2:40197)
I20260929 14:47:52.918085 7088 ts_manager.cc:194] Registered new tserver with Master: 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2:40197)
I20260929 14:47:52.920708 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.897414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41603 (local address 127.0.98.62:40107)
0929 14:47:52.897646 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.897653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.897672 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:47:52.897728 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.897733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.897794 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.897882 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:52.900210 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901110 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.905159 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.905505 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.905586 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.917005 (+ 11419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:52.917009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:52.917012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:52.917042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:52.918948 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.919452 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.919456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.919458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.919541 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:52.919825 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:52.919836 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:52.919838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:52.920031 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:52.920137 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.920441 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.920551 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 14:47:52.920732 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.897055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:41603)
0929 14:47:52.897528 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.897546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.897617 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.898157 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.898169 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.898979 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:52.899983 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.900005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.901302 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.901306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.905003 (+ 3697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.905014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.905763 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.905770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.905898 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.906525 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:52.906537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:52.916815 (+ 10278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:52.919135 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.919138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.919140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.919350 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.919641 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:52.919644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:52.919646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:52.919735 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:52.920135 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:52.920138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:52.920260 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.920379 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.920542 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":369,"spinlock_wait_cycles":21248,"thread_start_us":75,"threads_started":1}
I20260929 14:47:52.921571 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:52.921697 7594 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:52.921947 7594 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
I20260929 14:47:52.922690 6492 ts_manager.cc:194] Re-registered known tserver with Master: 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2:40197)
I20260929 14:47:52.923281 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41603
I20260929 14:47:52.950738 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.946027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:34012)
0929 14:47:52.946290 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:52.946309 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:52.946443 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:52.947019 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:52.947022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:52.947041 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:52.947140 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.947144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.948418 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.948421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.949211 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:52.949219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.950204 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.950207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:52.950381 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.950421 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:47:52.950494 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:47:52.950606 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260929 14:47:52.951107 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:52.946186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.0.98.60:35353)
0929 14:47:52.946370 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:52.946378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:52.946408 (+ 30us) server_negotiation.cc:407] Connection header received
0929 14:47:52.946586 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:52.946590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:52.946829 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:52.946988 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:52.947260 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.948090 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.949352 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:52.950060 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:52.950182 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:52.950350 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:52.950563 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 14:47:52.950895 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 14:47:52.951964 7287 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 14:47:52.952514 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c48ccb36f94260a3b4ed461872d21e. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:52.952704 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: LEARNER
W20260929 14:47:53.048604 7600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:53.048951 7600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:53.049039 7600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:53.053907 7600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:53.054010 7600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:53.054123 7600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:53.060614 7600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39917
--tserver_master_addrs=127.0.98.62:40107,127.0.98.61:42629,127.0.98.60:35353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:53.062143 7600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:53.063488 7600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:53.075667 7607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:53.075978 7600 server_base.cc:1034] running on GCE node
W20260929 14:47:53.075667 7609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:53.075659 7606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:53.076838 7600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:53.077652 7600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:53.078886 7600 hybrid_clock.cc:648] HybridClock initialized: now 1790693273078864 us; error 36 us; skew 500 ppm
Sep 29 14:47:53 dist-test-slave-5hd1 krb5kdc[6448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:53.082468 7600 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:47:53.083668 7600 webserver.cc:460] Webserver started at http://127.0.98.3:39917/ using document root <none> and password file <none>
I20260929 14:47:53.084473 7600 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:53.084602 7600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:53.089627 7600 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:53.092937 7616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:53.094578 7600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:47:53.094760 7600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
format_stamp: "Formatted at 2026-09-29 14:47:51 on dist-test-slave-5hd1"
server_key: "ede9d1b95d9dd6f0f25520c395d2840a"
server_key_iv: "ee7e49f1dd73eba0bed2d097ec63f116"
server_key_version: "encryptionkey@0"
I20260929 14:47:53.095494 7600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:53.111403 7600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:53.116403 7600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:53.116729 7600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:53.117647 7600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:47:53.118909 7600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:53.118968 7600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:53.119040 7600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:53.119076 7600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:53.133244 7600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43855
I20260929 14:47:53.133292 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43855 every 8 connection(s)
I20260929 14:47:53.134712 7600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:53.145234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7600
Sep 29 14:47:53 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 14:47:53 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:47:53 dist-test-slave-5hd1 krb5kdc[6448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693273, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:53.157052 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.137019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55205 (local address 127.0.98.61:42629)
0929 14:47:53.137215 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.137221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.138491 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 14:47:53.140360 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.140366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.140439 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.140536 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.142424 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.143318 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.144874 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145226 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145299 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.151804 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.151808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.151811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.151841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.153973 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.154703 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.154709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.154714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.154817 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155258 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.155261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.155263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.155537 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.155631 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.156735 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.156867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:47:53.157052 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.139210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39495 (local address 127.0.98.62:40107)
0929 14:47:53.139355 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.139358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.139801 (+ 443us) server_negotiation.cc:407] Connection header received
0929 14:47:53.140422 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.140425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.140475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.140533 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.142395 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.142992 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.144857 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145186 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145273 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.151675 (+ 6402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.151680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.151684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.151719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.154355 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155051 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.155055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.155057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.155151 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155470 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.155474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.155476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.155704 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.155789 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.156707 (+ 918us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.156855 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:47:53.158077 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.139042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:39495)
0929 14:47:53.139673 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.139692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.140084 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.140796 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.140805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.141424 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:53.142232 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.142254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.143142 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.143146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.144706 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.144718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145603 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.145834 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.146809 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:53.146843 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:53.151384 (+ 4541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:53.154475 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.154479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.154481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.154914 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.155256 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.155261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.155370 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.155827 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:53.155830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:53.156216 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.156443 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.157724 (+ 1281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":1272,"spinlock_wait_cycles":33024,"thread_start_us":114,"threads_started":1}
I20260929 14:47:53.158087 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.137624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.98.3:48543)
0929 14:47:53.138658 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.138671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.140031 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.140799 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.140808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.141389 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:53.142240 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.142253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.143472 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.143476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.144476 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.144486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145507 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.145824 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.146892 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:53.146909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:53.152131 (+ 5222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:53.154438 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.154444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.154448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.154905 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.155263 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.155268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.155417 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.155904 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:53.155907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:53.156207 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.156443 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.156885 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":261,"spinlock_wait_cycles":98944,"thread_start_us":199,"threads_started":1}
I20260929 14:47:53.158435 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.137625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.98.3:55205)
0929 14:47:53.138310 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.138360 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.140034 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.140799 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.140808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.141471 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:53.142240 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.142253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.143451 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.143454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.144711 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.144718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145504 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.145869 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.147014 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:53.147029 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:53.151529 (+ 4500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:53.154096 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.154108 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.154134 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.154574 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.154924 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:53.154928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:53.154932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:53.155144 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:53.155688 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:53.155697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:53.156181 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.156443 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.158250 (+ 1807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1754,"spinlock_wait_cycles":49408,"thread_start_us":745,"threads_started":1}
I20260929 14:47:53.159597 7732 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42629
I20260929 14:47:53.159920 7732 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:53.160045 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.137753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48543 (local address 127.0.98.60:35353)
0929 14:47:53.137902 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.137906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.138754 (+ 848us) server_negotiation.cc:407] Connection header received
0929 14:47:53.140469 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.140472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.140565 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.140649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.142417 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.143318 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.144613 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.145102 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.145245 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.152371 (+ 7126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.152375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.152378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.152412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.154303 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155049 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.155053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.155055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.155161 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.155517 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.155521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.155523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.155766 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.155860 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.159703 (+ 3843us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.159841 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:47:53.160708 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35353
I20260929 14:47:53.160765 7732 heartbeater.cc:511] Master 127.0.98.61:42629 requested a full tablet report, sending...
I20260929 14:47:53.160832 7730 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:53.161098 7730 heartbeater.cc:511] Master 127.0.98.60:35353 requested a full tablet report, sending...
I20260929 14:47:53.161499 7731 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40107
I20260929 14:47:53.161585 7731 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:53.161788 7731 heartbeater.cc:511] Master 127.0.98.62:40107 requested a full tablet report, sending...
I20260929 14:47:53.162173 7277 ts_manager.cc:194] Registered new tserver with Master: b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855)
I20260929 14:47:53.162695 6492 ts_manager.cc:194] Re-registered known tserver with Master: b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855)
I20260929 14:47:53.162845 7088 ts_manager.cc:194] Registered new tserver with Master: b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855)
I20260929 14:47:53.163654 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39495
I20260929 14:47:53.166639 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.148418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.98.61:42629)
0929 14:47:53.148698 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.148703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.149841 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 14:47:53.150035 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.150038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.150102 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.150267 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.151705 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.152679 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.159267 (+ 6588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.159661 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.159770 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.161986 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.161990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.161993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.162017 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.164552 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.165072 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.165075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.165078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.165179 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.165576 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.165579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.165581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.165801 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.165920 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.166295 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.166443 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260929 14:47:53.167127 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.147958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.98.62:40107)
0929 14:47:53.148332 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.148337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.148721 (+ 384us) server_negotiation.cc:407] Connection header received
0929 14:47:53.148934 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.148940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.149015 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.149087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.150211 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.151021 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.156750 (+ 5729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.157073 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.157146 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.163292 (+ 6146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.163296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.163298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.163325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.165077 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.165668 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.165671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.165673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.165761 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.166096 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.166102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.166105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.166340 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.166462 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.166817 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.166950 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260929 14:47:53.168612 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.150855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.98.60:35353)
0929 14:47:53.152920 (+ 2065us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.152925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.152938 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:47:53.152975 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.152978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.153030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.153135 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.155884 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.156606 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.158690 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.159002 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.159094 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.162885 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.162888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.162891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.162916 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.166177 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.166667 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.166674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.166678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.166795 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.167120 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.167123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.167125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.167509 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.167691 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.168328 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.168480 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":105,"threads_started":1}
I20260929 14:47:53.170614 6492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:53.174073 6492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:53.177631 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: true } } }
I20260929 14:47:53.177842 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:53.181861 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: attempting to promote NON_VOTER b899fad8922247849d3853c95394b98c to VOTER
I20260929 14:47:53.182296 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } }
I20260929 14:47:53.183880 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:47:53.183990 7287 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 LEARNER]: Refusing update from remote peer e2c48ccb36f94260a3b4ed461872d21e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:47:53.184361 7748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:47:53.184713 7748 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:47:53.186030 7749 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b899fad8922247849d3853c95394b98c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.186398 7097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b899fad8922247849d3853c95394b98c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.187655 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.187781 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:53.187908 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.187991 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:53.188728 7757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.188910 7757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:53.189597 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.189723 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:53.190244 7287 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b899fad8922247849d3853c95394b98c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.193626 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2c48ccb36f94260a3b4ed461872d21e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:53.193771 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:53.199115 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.192354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43855 (local address 127.0.0.1:53238)
0929 14:47:53.193222 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.193235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.193624 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.194197 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.194200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.194217 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:53.194612 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.194624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.196538 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.196542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.197631 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.197640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.198571 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.198574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.198678 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.198712 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.198796 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.198900 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":127,"threads_started":1}
I20260929 14:47:53.199424 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.192601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.0.98.3:43855)
0929 14:47:53.193072 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.193086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.193735 (+ 649us) server_negotiation.cc:407] Connection header received
0929 14:47:53.193843 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.193849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.194056 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.194226 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:53.194762 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.195858 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.197759 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.198427 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.198541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.198696 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.198845 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.198942 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":104,"threads_started":1}
I20260929 14:47:53.200243 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.191974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40197 (local address 127.0.0.1:44488)
0929 14:47:53.192125 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.192141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.192294 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.194982 (+ 2688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.194986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.195006 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:53.195129 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.195136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.196862 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.196865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.198107 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.198119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.199797 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.199801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.199890 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.199910 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.199981 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.200076 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 14:47:53.200486 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.192304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.98.2:40197)
0929 14:47:53.194397 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.194408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.194435 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:47:53.194543 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.194550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.194808 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.195016 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:53.195384 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.196705 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.198517 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.199513 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.199803 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.199991 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.200092 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.200229 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":163,"threads_started":1}
I20260929 14:47:53.201268 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.192993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42085 (local address 127.0.0.1:54818)
0929 14:47:53.194587 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.194599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.194686 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.195286 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.195289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.195305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:53.195386 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.195391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.197030 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.197033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.198187 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.198197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.200859 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.200862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.200955 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.200978 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.201062 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.201146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":80,"threads_started":1}
I20260929 14:47:53.201601 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.193347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.98.1:42085)
0929 14:47:53.194468 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.194474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.194780 (+ 306us) server_negotiation.cc:407] Connection header received
0929 14:47:53.194856 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.194861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.195021 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.195153 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:53.195541 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.196738 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.198341 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.199177 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.200851 (+ 1674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.201003 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.201107 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.201209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":74,"threads_started":1}
I20260929 14:47:53.202837 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27049a99064457806b0464ce84aeab (DEFAULT_TABLE table=test-table [id=5b04bdc5af5e4a32b1ba8c023337662e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:53.204085 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27049a99064457806b0464ce84aeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:53.203843 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27049a99064457806b0464ce84aeab (DEFAULT_TABLE table=test-table [id=5b04bdc5af5e4a32b1ba8c023337662e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:53.205245 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27049a99064457806b0464ce84aeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:53.205842 7390 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27049a99064457806b0464ce84aeab (DEFAULT_TABLE table=test-table [id=5b04bdc5af5e4a32b1ba8c023337662e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:53.207084 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27049a99064457806b0464ce84aeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:53.213577 7763 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Bootstrap starting.
I20260929 14:47:53.216946 7765 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Bootstrap starting.
I20260929 14:47:53.217010 7764 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Bootstrap starting.
I20260929 14:47:53.217322 7763 tablet_bootstrap.cc:654] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:53.218464 7763 log.cc:824] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:53.219326 7765 tablet_bootstrap.cc:654] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:53.219925 7764 tablet_bootstrap.cc:654] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:53.220227 7765 log.cc:824] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:53.220844 7764 log.cc:824] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:53.221475 7763 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: No bootstrap required, opened a new log
I20260929 14:47:53.221751 7763 ts_tablet_manager.cc:1397] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:47:53.222340 7765 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: No bootstrap required, opened a new log
I20260929 14:47:53.222549 7765 ts_tablet_manager.cc:1397] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:47:53.223104 7764 tablet_bootstrap.cc:492] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: No bootstrap required, opened a new log
I20260929 14:47:53.223299 7764 ts_tablet_manager.cc:1397] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 14:47:53.225759 7765 raft_consensus.cc:348] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.226025 7765 raft_consensus.cc:374] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:53.226087 7765 raft_consensus.cc:729] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8681f175c4e549a487c8eda1ed0a934d, State: Initialized, Role: FOLLOWER
I20260929 14:47:53.226588 7765 consensus_queue.cc:260] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.227319 7457 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:53.227394 7763 raft_consensus.cc:348] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.227742 7763 raft_consensus.cc:374] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:53.227802 7763 raft_consensus.cc:729] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a0c2cf5c944ee085ac8eb776d833e6, State: Initialized, Role: FOLLOWER
I20260929 14:47:53.228369 7763 consensus_queue.cc:260] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.229022 7765 ts_tablet_manager.cc:1428] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:47:53.231774 7763 ts_tablet_manager.cc:1428] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260929 14:47:53.232277 7731 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:53.232318 7764 raft_consensus.cc:348] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.234440 7764 raft_consensus.cc:374] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:53.234509 7764 raft_consensus.cc:729] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f8a8f00b9d463698757b59bc4cb205, State: Initialized, Role: FOLLOWER
I20260929 14:47:53.235128 7764 consensus_queue.cc:260] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.236521 7764 ts_tablet_manager.cc:1428] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260929 14:47:53.236792 7594 heartbeater.cc:503] Master 127.0.98.62:40107 was elected leader, sending a full tablet report...
I20260929 14:47:53.377046 7771 raft_consensus.cc:482] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:53.377264 7771 raft_consensus.cc:504] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:53.378449 7771 leader_election.cc:288] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855), 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:53.383102 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.379040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39275 (local address 127.0.98.1:42085)
0929 14:47:53.379172 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.379177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.379199 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:53.379410 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.379416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.379477 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.379570 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:53.379964 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.380623 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.381924 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.382389 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.382476 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.382531 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.382865 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.382966 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:47:53.383165 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.378936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42085 (local address 127.0.98.2:39275)
0929 14:47:53.379112 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.379132 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.379262 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.379617 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.379620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.379639 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:53.379828 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.379836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.380756 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.380760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.381769 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.381779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.382558 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.382563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.382704 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.382736 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.382823 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.382920 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":54}
I20260929 14:47:53.383212 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.378936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43855 (local address 127.0.98.2:54781)
0929 14:47:53.379339 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:53.379359 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:53.379489 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:53.379812 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:53.379814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:53.379829 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:53.379972 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.379980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.380996 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.380999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.381903 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:53.381916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.382735 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.382738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:53.382849 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.382883 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:47:53.382977 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 14:47:53.383062 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":161}
I20260929 14:47:53.383277 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.379040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54781 (local address 127.0.98.3:43855)
0929 14:47:53.379172 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.379177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.379500 (+ 323us) server_negotiation.cc:407] Connection header received
0929 14:47:53.379619 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.379622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.379682 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.379777 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:53.380136 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.380836 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.382065 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.382566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.382707 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.382785 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.383016 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.383131 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:47:53.384078 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27049a99064457806b0464ce84aeab" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" is_pre_election: true
I20260929 14:47:53.384090 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27049a99064457806b0464ce84aeab" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8681f175c4e549a487c8eda1ed0a934d" is_pre_election: true
I20260929 14:47:53.384473 7410 raft_consensus.cc:2454] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 0.
I20260929 14:47:53.384490 7684 raft_consensus.cc:2454] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 0.
I20260929 14:47:53.385016 7481 leader_election.cc:302] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36f8a8f00b9d463698757b59bc4cb205, 8681f175c4e549a487c8eda1ed0a934d, b6a0c2cf5c944ee085ac8eb776d833e6; no voters:
I20260929 14:47:53.385375 7771 raft_consensus.cc:2781] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:53.385475 7771 raft_consensus.cc:482] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:53.385509 7771 raft_consensus.cc:3037] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:53.386883 7771 raft_consensus.cc:504] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
W20260929 14:47:53.386934 7733 tablet.cc:2369] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:53.387362 7771 leader_election.cc:288] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 election: Requested vote from peers b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855), 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:53.387785 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27049a99064457806b0464ce84aeab" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
I20260929 14:47:53.387780 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27049a99064457806b0464ce84aeab" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8681f175c4e549a487c8eda1ed0a934d"
I20260929 14:47:53.387933 7684 raft_consensus.cc:3037] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:53.387933 7410 raft_consensus.cc:3037] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:53.389343 7410 raft_consensus.cc:2454] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 1.
I20260929 14:47:53.389343 7684 raft_consensus.cc:2454] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 1.
I20260929 14:47:53.389741 7482 leader_election.cc:302] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36f8a8f00b9d463698757b59bc4cb205, 8681f175c4e549a487c8eda1ed0a934d, b6a0c2cf5c944ee085ac8eb776d833e6; no voters:
I20260929 14:47:53.389959 7771 raft_consensus.cc:2781] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:53.390208 7771 raft_consensus.cc:686] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 1 LEADER]: Becoming Leader. State: Replica: 36f8a8f00b9d463698757b59bc4cb205, State: Running, Role: LEADER
I20260929 14:47:53.390674 7771 consensus_queue.cc:237] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
W20260929 14:47:53.392270 7596 tablet.cc:2369] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:53.395177 6492 catalog_manager.cc:5495] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "36f8a8f00b9d463698757b59bc4cb205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20260929 14:47:53.412549 7459 tablet.cc:2369] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:53.440766 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.436955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.0.98.2:40197)
0929 14:47:53.437122 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.437127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.437147 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:47:53.437318 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.437321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.437381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.437501 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:53.437778 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.438302 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.439147 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.439414 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.439530 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.439847 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:53.440371 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:53.440506 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.440547 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.440601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:47:53.447819 7684 raft_consensus.cc:1261] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Refusing update from remote peer 36f8a8f00b9d463698757b59bc4cb205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:53.447880 7410 raft_consensus.cc:1261] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Refusing update from remote peer 36f8a8f00b9d463698757b59bc4cb205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:53.448788 7771 consensus_queue.cc:1035] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:53.449069 7772 consensus_queue.cc:1035] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:53.455979 7775 mvcc.cc:205] Tried to move back new op lower bound from 7334679648034050048 to 7334679647811973120. Current Snapshot: MvccSnapshot[applied={T|T < 7334679648034050048}]
I20260929 14:47:53.465736 7778 mvcc.cc:205] Tried to move back new op lower bound from 7334679648034050048 to 7334679647811973120. Current Snapshot: MvccSnapshot[applied={T|T < 7334679648034050048}]
I20260929 14:47:53.469274 6492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:53.469534 6492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:53.475831 6492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c48ccb36f94260a3b4ed461872d21e: Sending DeleteTablet for 3 replicas of tablet 4d27049a99064457806b0464ce84aeab
I20260929 14:47:53.476787 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d27049a99064457806b0464ce84aeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44488
I20260929 14:47:53.476989 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d27049a99064457806b0464ce84aeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260929 14:47:53.477516 7785 tablet_replica.cc:330] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: stopping tablet replica
I20260929 14:47:53.477651 7786 tablet_replica.cc:330] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: stopping tablet replica
I20260929 14:47:53.477612 7390 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d27049a99064457806b0464ce84aeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54818
I20260929 14:47:53.478050 7785 raft_consensus.cc:2229] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:53.478050 7786 raft_consensus.cc:2229] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:53.478468 7786 raft_consensus.cc:2258] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:53.478598 7785 raft_consensus.cc:2258] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:53.480458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6461
I20260929 14:47:53.481503 7786 ts_tablet_manager.cc:1905] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:53.482133 7785 ts_tablet_manager.cc:1905] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:53.485095 7786 ts_tablet_manager.cc:1918] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:53.485244 7786 log.cc:1196] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/4d27049a99064457806b0464ce84aeab
I20260929 14:47:53.485638 7786 ts_tablet_manager.cc:1939] T 4d27049a99064457806b0464ce84aeab P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting consensus metadata
I20260929 14:47:53.485608 7785 ts_tablet_manager.cc:1918] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:53.485846 7785 log.cc:1196] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/4d27049a99064457806b0464ce84aeab
I20260929 14:47:53.486232 7785 ts_tablet_manager.cc:1939] T 4d27049a99064457806b0464ce84aeab P 36f8a8f00b9d463698757b59bc4cb205: Deleting consensus metadata
I20260929 14:47:53.489117 7787 tablet_replica.cc:330] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: stopping tablet replica
W20260929 14:47:53.489429 7619 connection.cc:535] server connection from 127.0.0.1:53238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 14:47:53.489869 7483 connection.cc:535] server connection from 127.0.0.1:44488 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:47:53.490198 7787 raft_consensus.cc:2229] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:53.490739 7787 raft_consensus.cc:2258] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:53.492914 7787 ts_tablet_manager.cc:1905] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:53.496810 7787 ts_tablet_manager.cc:1918] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:53.497027 7787 log.cc:1196] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/4d27049a99064457806b0464ce84aeab
I20260929 14:47:53.497527 7787 ts_tablet_manager.cc:1939] T 4d27049a99064457806b0464ce84aeab P 8681f175c4e549a487c8eda1ed0a934d: Deleting consensus metadata
I20260929 14:47:53.499356 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.479806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.0.98.61:42629)
0929 14:47:53.479956 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.479962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.488688 (+ 8726us) server_negotiation.cc:407] Connection header received
0929 14:47:53.488898 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.488902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.488990 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.489075 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.490238 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.491052 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.492243 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.492570 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.492652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.495070 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.495077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.495083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.495118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.497109 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.497676 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.497679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.497681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.497769 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.498032 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.498035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.498037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.498267 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.498387 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.499039 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.499196 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 14:47:53.499547 7345 connection.cc:441] server connection from 127.0.0.1:54818 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54818 (request call id 1) could send its response
I20260929 14:47:53.509433 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.494948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.0.98.61:42629)
0929 14:47:53.495057 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.495061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.496490 (+ 1429us) server_negotiation.cc:407] Connection header received
0929 14:47:53.497868 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.497871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.497920 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.497984 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.499769 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.500603 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.501728 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.502108 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.502219 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.504803 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.504808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.504812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.504839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.507145 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.507739 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.507743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.507745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.507840 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.508483 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.508487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.508489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.508699 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.508826 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.509146 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.509278 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:47:53.510958 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.479930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.0.98.60:35353)
0929 14:47:53.480046 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.480051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.500108 (+ 20057us) server_negotiation.cc:407] Connection header received
0929 14:47:53.500318 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.500322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.500371 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.500420 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.501364 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.502172 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.503952 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.504324 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.504400 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.506017 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.506021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.506024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.506048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.508798 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.509298 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.509302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.509304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.509453 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.509767 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.509774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.509779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.509988 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.510100 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.510563 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.510751 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":5248}
I20260929 14:47:53.511348 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:53.494347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.0.98.60:35353)
0929 14:47:53.494459 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:53.494463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:53.495946 (+ 1483us) server_negotiation.cc:407] Connection header received
0929 14:47:53.496582 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:53.496588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:53.496662 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:53.496728 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:53.498773 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.499579 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.502853 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:53.503176 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:53.503326 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:53.504995 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:53.505014 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:53.505018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:53.505049 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:53.508355 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.508871 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.508874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.508876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.508966 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:53.509298 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:53.509302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:53.509304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:53.509620 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:53.509758 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:53.510563 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 14:47:53.511236 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":512,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":8960}
I20260929 14:47:54.463781 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.463200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:34139)
0929 14:47:54.463502 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.463632 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
W20260929 14:47:54.464223 7731 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.465519 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.465061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:39607)
0929 14:47:54.465331 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.465428 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
W20260929 14:47:54.465917 7457 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.466637 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.466248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:56563)
0929 14:47:54.466487 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.466559 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
W20260929 14:47:54.466918 7594 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.974978 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2c48ccb36f94260a3b4ed461872d21e)
I20260929 14:47:54.975157 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } }
I20260929 14:47:54.975796 7799 leader_election.cc:288] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107), 56238d34d13743fb89b379b18e888f72 (127.0.98.61:42629)
I20260929 14:47:54.976441 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.975939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58716)
0929 14:47:54.976231 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.976337 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":61,"threads_started":1}
I20260929 14:47:54.976418 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2c48ccb36f94260a3b4ed461872d21e)
W20260929 14:47:54.976681 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.976939 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } }
I20260929 14:47:54.977643 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.977455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58718)
0929 14:47:54.977541 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.977585 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 14:47:54.977859 7262 leader_election.cc:334] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.978415 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107), b899fad8922247849d3853c95394b98c (127.0.98.60:35353)
I20260929 14:47:54.978965 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.978514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58732)
0929 14:47:54.978739 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.978846 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
W20260929 14:47:54.979171 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.980135 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.979822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58748)
0929 14:47:54.979980 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.980042 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 14:47:54.980312 7080 leader_election.cc:334] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.981182 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.976185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42629 (local address 127.0.0.1:60872)
0929 14:47:54.976628 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.976648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:54.976791 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:54.977159 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:54.977162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:54.977184 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:54.977330 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:54.977341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.978438 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.978445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:54.979759 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:54.979771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.980702 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.980705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:54.980799 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:54.980823 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:47:54.980962 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 14:47:54.981043 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":71,"threads_started":1}
I20260929 14:47:54.981168 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.976362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.0.98.61:42629)
0929 14:47:54.976743 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:54.976749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:54.976771 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:54.976931 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:54.976937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:54.977007 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:54.977135 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:54.977488 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.978307 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.979861 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.980574 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.980690 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:54.980762 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:54.980960 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 14:47:54.981041 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":290,"thread_start_us":106,"threads_started":1}
I20260929 14:47:54.981707 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b899fad8922247849d3853c95394b98c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56238d34d13743fb89b379b18e888f72" is_pre_election: true
I20260929 14:47:54.981909 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b899fad8922247849d3853c95394b98c in term 1.
I20260929 14:47:54.982266 7261 leader_election.cc:302] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56238d34d13743fb89b379b18e888f72, b899fad8922247849d3853c95394b98c; no voters: e2c48ccb36f94260a3b4ed461872d21e
I20260929 14:47:54.982445 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:47:54.982511 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Starting leader election (detected failure of leader e2c48ccb36f94260a3b4ed461872d21e)
I20260929 14:47:54.982569 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:47:54.983443 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.978852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35353 (local address 127.0.0.1:34038)
0929 14:47:54.979549 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.979563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:54.979662 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:54.979932 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:54.979935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:54.979953 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:54.980033 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:54.980039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.980979 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.980984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:54.982165 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:54.982175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.983004 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.983008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:54.983099 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:54.983130 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 14:47:54.983218 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 14:47:54.983303 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":53,"threads_started":1}
I20260929 14:47:54.983443 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.979055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.0.98.60:35353)
0929 14:47:54.979342 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:54.979357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:54.979657 (+ 300us) server_negotiation.cc:407] Connection header received
0929 14:47:54.979764 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:54.979768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:54.979822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:54.979925 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:54.980160 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.980858 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.982268 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:54.982877 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:54.982998 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:54.983063 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:54.983244 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 14:47:54.983328 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260929 14:47:54.983777 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } }
I20260929 14:47:54.984225 7799 leader_election.cc:288] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 election: Requested vote from peers e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107), 56238d34d13743fb89b379b18e888f72 (127.0.98.61:42629)
I20260929 14:47:54.984225 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56238d34d13743fb89b379b18e888f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b899fad8922247849d3853c95394b98c" is_pre_election: true
I20260929 14:47:54.984586 7287 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56238d34d13743fb89b379b18e888f72 in current term 2: Already voted for candidate b899fad8922247849d3853c95394b98c in this term.
I20260929 14:47:54.984684 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.984346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58758)
0929 14:47:54.984514 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.984564 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 14:47:54.985070 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b899fad8922247849d3853c95394b98c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56238d34d13743fb89b379b18e888f72"
I20260929 14:47:54.985117 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56238d34d13743fb89b379b18e888f72; no voters: b899fad8922247849d3853c95394b98c, e2c48ccb36f94260a3b4ed461872d21e
I20260929 14:47:54.985209 7097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:47:54.986495 7097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b899fad8922247849d3853c95394b98c in term 2.
I20260929 14:47:54.986584 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:47:54.986830 7261 leader_election.cc:302] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56238d34d13743fb89b379b18e888f72, b899fad8922247849d3853c95394b98c; no voters:
I20260929 14:47:54.986968 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:54.985392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58760)
0929 14:47:54.986743 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:54.986812 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1283}
I20260929 14:47:54.987075 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 2 FOLLOWER]: Leader election won for term 2
W20260929 14:47:54.987133 7262 leader_election.cc:334] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
I20260929 14:47:54.987318 7799 raft_consensus.cc:686] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [term 2 LEADER]: Becoming Leader. State: Replica: b899fad8922247849d3853c95394b98c, State: Running, Role: LEADER
I20260929 14:47:54.987675 7799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } }
I20260929 14:47:54.989562 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b899fad8922247849d3853c95394b98c. Latest consensus state: current_term: 2 leader_uuid: "b899fad8922247849d3853c95394b98c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:54.989678 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:54.990020 7811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:54.992839 7811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:54.993244 7811 catalog_manager.cc:1259] Loaded cluster ID: 2f615b46235e411b87ca818eb10de484
I20260929 14:47:54.993291 7811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:54.993959 7811 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:54.994410 7811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Loaded TSK: 0
I20260929 14:47:54.995126 7811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:47:55.050086 7277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:55.054154 7277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:55.057817 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.057525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58776)
0929 14:47:55.057663 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.057733 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 14:47:55.057902 7097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [term 2 FOLLOWER]: Refusing update from remote peer b899fad8922247849d3853c95394b98c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 14:47:55.058553 7809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 14:47:55.058805 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.058497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.0.1:58790)
0929 14:47:55.058623 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.058691 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 14:47:55.058940 7262 consensus_peers.cc:487] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c -> Peer e2c48ccb36f94260a3b4ed461872d21e (127.0.98.62:40107): Couldn't send request to peer e2c48ccb36f94260a3b4ed461872d21e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:47:55.060437 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b899fad8922247849d3853c95394b98c. Latest consensus state: current_term: 2 leader_uuid: "b899fad8922247849d3853c95394b98c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:55.060621 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:55.061962 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b899fad8922247849d3853c95394b98c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:55.062106 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:55.062408 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b899fad8922247849d3853c95394b98c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e2c48ccb36f94260a3b4ed461872d21e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40107 } } peers { permanent_uuid: "56238d34d13743fb89b379b18e888f72" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42629 } attrs { promote: false } } peers { permanent_uuid: "b899fad8922247849d3853c95394b98c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35353 } attrs { promote: false } } }
I20260929 14:47:55.062546 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56238d34d13743fb89b379b18e888f72 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:47:55.062762 7813 mvcc.cc:205] Tried to move back new op lower bound from 7334679654631591936 to 7334679654352773120. Current Snapshot: MvccSnapshot[applied={T|T < 7334679654631591936}]
I20260929 14:47:55.074803 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.070366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43855 (local address 127.0.0.1:53242)
0929 14:47:55.070528 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.070541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:55.070643 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:55.071427 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:55.071431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:55.071451 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:55.071545 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.071551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.072761 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.072767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.073606 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.073615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.074337 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.074340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.074420 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.074447 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:47:55.074505 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:47:55.074581 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":6,"spinlock_wait_cycles":8960}
I20260929 14:47:55.074868 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.069767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40197 (local address 127.0.0.1:44510)
0929 14:47:55.069925 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.069944 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:55.070044 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:55.070600 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:55.070604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:55.070643 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:55.070790 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.070799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.071983 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.071987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.073307 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.073315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.074272 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.074276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.074374 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.074407 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 14:47:55.074484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 14:47:55.074581 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":1,"spinlock_wait_cycles":2432}
I20260929 14:47:55.075492 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.070506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.0.98.3:43855)
0929 14:47:55.071162 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:55.071167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:55.071181 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:55.071234 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:55.071237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:55.071287 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:55.071388 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:55.071878 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.072645 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.073732 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.074182 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.075104 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.075189 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:55.075227 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 14:47:55.075337 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":162,"threads_started":1}
I20260929 14:47:55.075839 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.069963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.0.98.2:40197)
0929 14:47:55.070228 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:55.070234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:55.070254 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:47:55.070319 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:55.070325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:55.070383 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:55.070914 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:55.071034 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.071857 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.073459 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.074135 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.074250 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.074322 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:55.074678 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 14:47:55.074796 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260929 14:47:55.076152 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab37c87dd81416a863a0be056741878 (DEFAULT_TABLE table=test-table [id=370ec651425443438f98a47c8d4d1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:55.076489 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.070411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42085 (local address 127.0.0.1:54826)
0929 14:47:55.070898 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.070913 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:55.070991 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:55.071701 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:55.071705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:55.071720 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:55.071789 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.071794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.072982 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.072985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.074330 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:55.074342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.076018 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.076022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:55.076114 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.076138 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:47:55.076252 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:47:55.076344 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":119,"threads_started":1}
I20260929 14:47:55.076534 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab37c87dd81416a863a0be056741878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:55.076742 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab37c87dd81416a863a0be056741878 (DEFAULT_TABLE table=test-table [id=370ec651425443438f98a47c8d4d1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:55.076956 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.070871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.98.1:42085)
0929 14:47:55.071136 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:55.071141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:55.071156 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:47:55.071223 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:55.071226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:55.071281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:55.071388 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:55.071942 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.072841 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.074757 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.075765 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.075885 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.075955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:55.076664 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 14:47:55.076784 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260929 14:47:55.077131 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab37c87dd81416a863a0be056741878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:55.078933 7822 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Bootstrap starting.
I20260929 14:47:55.079952 7823 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Bootstrap starting.
I20260929 14:47:55.080036 7822 tablet_bootstrap.cc:654] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:55.080320 7390 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab37c87dd81416a863a0be056741878 (DEFAULT_TABLE table=test-table [id=370ec651425443438f98a47c8d4d1b0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:55.080639 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab37c87dd81416a863a0be056741878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:55.081023 7823 tablet_bootstrap.cc:654] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:55.082252 7822 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: No bootstrap required, opened a new log
I20260929 14:47:55.082268 7823 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: No bootstrap required, opened a new log
I20260929 14:47:55.082358 7823 ts_tablet_manager.cc:1397] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:55.082357 7822 ts_tablet_manager.cc:1397] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:55.082819 7823 raft_consensus.cc:348] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.082819 7822 raft_consensus.cc:348] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.082979 7822 raft_consensus.cc:374] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:55.082979 7823 raft_consensus.cc:374] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:55.083022 7822 raft_consensus.cc:729] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a0c2cf5c944ee085ac8eb776d833e6, State: Initialized, Role: FOLLOWER
I20260929 14:47:55.083022 7823 raft_consensus.cc:729] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f8a8f00b9d463698757b59bc4cb205, State: Initialized, Role: FOLLOWER
I20260929 14:47:55.083180 7823 consensus_queue.cc:260] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.083204 7822 consensus_queue.cc:260] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.083588 7823 ts_tablet_manager.cc:1428] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:55.083588 7822 ts_tablet_manager.cc:1428] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:55.085274 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.084467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:44055)
0929 14:47:55.085087 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.085154 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":101,"threads_started":1}
I20260929 14:47:55.085887 7826 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d: Bootstrap starting.
I20260929 14:47:55.087287 7826 tablet_bootstrap.cc:654] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:55.087672 7730 heartbeater.cc:503] Master 127.0.98.60:35353 was elected leader, sending a full tablet report...
I20260929 14:47:55.088208 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.084466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:40483)
0929 14:47:55.087977 (+ 3511us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.088047 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3420,"thread_start_us":101,"threads_started":1}
I20260929 14:47:55.088810 7826 tablet_bootstrap.cc:492] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d: No bootstrap required, opened a new log
I20260929 14:47:55.088935 7826 ts_tablet_manager.cc:1397] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:55.089149 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.088648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:41651)
0929 14:47:55.088953 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.089057 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":119,"threads_started":1}
I20260929 14:47:55.089552 7826 raft_consensus.cc:348] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.089713 7826 raft_consensus.cc:374] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:55.089747 7826 raft_consensus.cc:729] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8681f175c4e549a487c8eda1ed0a934d, State: Initialized, Role: FOLLOWER
I20260929 14:47:55.089922 7826 consensus_queue.cc:260] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.090371 7826 ts_tablet_manager.cc:1428] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:47:55.091038 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.090643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:34067)
0929 14:47:55.090881 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.090936 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
I20260929 14:47:55.091864 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.091616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:45903)
0929 14:47:55.091737 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.091795 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 14:47:55.092476 7456 heartbeater.cc:503] Master 127.0.98.60:35353 was elected leader, sending a full tablet report...
I20260929 14:47:55.169454 7828 raft_consensus.cc:482] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:55.169687 7828 raft_consensus.cc:504] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.170293 7828 leader_election.cc:288] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855), 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:55.170892 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab37c87dd81416a863a0be056741878" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" is_pre_election: true
I20260929 14:47:55.170962 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab37c87dd81416a863a0be056741878" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8681f175c4e549a487c8eda1ed0a934d" is_pre_election: true
I20260929 14:47:55.171097 7684 raft_consensus.cc:2454] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 0.
I20260929 14:47:55.171203 7410 raft_consensus.cc:2454] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 0.
I20260929 14:47:55.171509 7481 leader_election.cc:302] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f8a8f00b9d463698757b59bc4cb205, b6a0c2cf5c944ee085ac8eb776d833e6; no voters:
I20260929 14:47:55.171775 7828 raft_consensus.cc:2781] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:55.171850 7828 raft_consensus.cc:482] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:55.171893 7828 raft_consensus.cc:3037] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:55.173287 7828 raft_consensus.cc:504] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.173702 7828 leader_election.cc:288] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 election: Requested vote from peers b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855), 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085)
I20260929 14:47:55.174153 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab37c87dd81416a863a0be056741878" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8681f175c4e549a487c8eda1ed0a934d"
I20260929 14:47:55.174314 7410 raft_consensus.cc:3037] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:55.174218 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab37c87dd81416a863a0be056741878" candidate_uuid: "36f8a8f00b9d463698757b59bc4cb205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6"
I20260929 14:47:55.174414 7684 raft_consensus.cc:3037] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:55.175668 7684 raft_consensus.cc:2454] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 1.
I20260929 14:47:55.175670 7410 raft_consensus.cc:2454] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f8a8f00b9d463698757b59bc4cb205 in term 1.
I20260929 14:47:55.176047 7481 leader_election.cc:302] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 36f8a8f00b9d463698757b59bc4cb205, 8681f175c4e549a487c8eda1ed0a934d, b6a0c2cf5c944ee085ac8eb776d833e6; no voters:
I20260929 14:47:55.176347 7828 raft_consensus.cc:2781] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:55.176645 7828 raft_consensus.cc:686] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 1 LEADER]: Becoming Leader. State: Replica: 36f8a8f00b9d463698757b59bc4cb205, State: Running, Role: LEADER
I20260929 14:47:55.176985 7593 heartbeater.cc:503] Master 127.0.98.60:35353 was elected leader, sending a full tablet report...
I20260929 14:47:55.176837 7828 consensus_queue.cc:237] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } }
I20260929 14:47:55.177749 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.177484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:41847)
0929 14:47:55.177613 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.177679 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 14:47:55.179338 7277 catalog_manager.cc:5495] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f8a8f00b9d463698757b59bc4cb205 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "36f8a8f00b9d463698757b59bc4cb205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f8a8f00b9d463698757b59bc4cb205" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:55.198405 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.195098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.0.98.2:40197)
0929 14:47:55.195308 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:55.195314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:55.195329 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:47:55.195488 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:55.195491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:55.195549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:55.195628 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:55.195878 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.196486 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.197348 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:55.197566 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:55.197675 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:55.197918 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:55.197990 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:55.198067 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:55.198187 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 14:47:55.198247 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 14:47:55.201208 7410 raft_consensus.cc:1261] T 3ab37c87dd81416a863a0be056741878 P 8681f175c4e549a487c8eda1ed0a934d [term 1 FOLLOWER]: Refusing update from remote peer 36f8a8f00b9d463698757b59bc4cb205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:55.201208 7684 raft_consensus.cc:1261] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Refusing update from remote peer 36f8a8f00b9d463698757b59bc4cb205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:55.201789 7828 consensus_queue.cc:1035] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a0c2cf5c944ee085ac8eb776d833e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:55.202003 7835 consensus_queue.cc:1035] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8681f175c4e549a487c8eda1ed0a934d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:55.204005 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.203693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:38491)
0929 14:47:55.203830 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.203918 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:47:55.204700 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.204395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:55619)
0929 14:47:55.204554 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.204620 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 14:47:55.207188 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.206834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:48209)
0929 14:47:55.207053 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.207108 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 14:47:55.207515 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.207198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.1:38729)
0929 14:47:55.207350 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.207416 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 14:47:55.211638 7276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:55.211815 7276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:55.212687 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.211638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.2:33909)
0929 14:47:55.212351 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.212581 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625}
I20260929 14:47:55.214300 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.213989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40107 (local address 127.0.98.3:58307)
0929 14:47:55.214125 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.214198 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:47:55.217967 7276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b899fad8922247849d3853c95394b98c: Sending DeleteTablet for 3 replicas of tablet 3ab37c87dd81416a863a0be056741878
I20260929 14:47:55.218792 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab37c87dd81416a863a0be056741878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53242
I20260929 14:47:55.218919 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab37c87dd81416a863a0be056741878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53242
I20260929 14:47:55.219537 7847 tablet_replica.cc:330] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: stopping tablet replica
I20260929 14:47:55.219681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7239
I20260929 14:47:55.219686 7847 raft_consensus.cc:2229] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:55.219532 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab37c87dd81416a863a0be056741878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44510
I20260929 14:47:55.220350 7847 raft_consensus.cc:2258] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:55.221235 7847 ts_tablet_manager.cc:1905] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:55.224413 7847 ts_tablet_manager.cc:1918] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:55.224516 7847 log.cc:1196] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/3ab37c87dd81416a863a0be056741878
I20260929 14:47:55.224768 7848 tablet_replica.cc:330] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: stopping tablet replica
I20260929 14:47:55.224802 7847 ts_tablet_manager.cc:1939] T 3ab37c87dd81416a863a0be056741878 P b6a0c2cf5c944ee085ac8eb776d833e6: Deleting consensus metadata
I20260929 14:47:55.224982 7848 raft_consensus.cc:2229] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:55.225291 7848 raft_consensus.cc:2258] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:55.225711 7262 catalog_manager.cc:4841] TS b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855): tablet 3ab37c87dd81416a863a0be056741878 (table test-table [id=370ec651425443438f98a47c8d4d1b0c]) successfully deleted
I20260929 14:47:55.226151 7848 ts_tablet_manager.cc:1905] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:47:55.226521 7262 catalog_manager.cc:4808] TS b6a0c2cf5c944ee085ac8eb776d833e6 (127.0.98.3:43855): delete failed for tablet 3ab37c87dd81416a863a0be056741878 because the tablet was not found. No further retry: Not found: Tablet not found: 3ab37c87dd81416a863a0be056741878
I20260929 14:47:55.229115 7848 ts_tablet_manager.cc:1918] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:55.229233 7848 log.cc:1196] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/3ab37c87dd81416a863a0be056741878
W20260929 14:47:55.229256 7263 connection.cc:535] client connection to 127.0.98.1:42085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:47:55.229607 7848 ts_tablet_manager.cc:1939] T 3ab37c87dd81416a863a0be056741878 P 36f8a8f00b9d463698757b59bc4cb205: Deleting consensus metadata
I20260929 14:47:55.229751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7464
I20260929 14:47:55.230695 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.229877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42085 (local address 127.0.0.1:54842)
0929 14:47:55.230517 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.230580 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525}
W20260929 14:47:55.237682 7263 catalog_manager.cc:4584] TS 8681f175c4e549a487c8eda1ed0a934d (127.0.98.1:42085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ab37c87dd81416a863a0be056741878: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42085: connect: Connection refused (error 111)
I20260929 14:47:55.238135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7600
I20260929 14:47:55.238265 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:55.237808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40197 (local address 127.0.0.1:44526)
0929 14:47:55.238038 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:55.238108 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 14:47:55.244359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 6957
I20260929 14:47:55.250965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7059
2026-09-29T14:47:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5868 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:55 dist-test-slave-5hd1 krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:55 dist-test-slave-5hd1 krb5kdc[7851](info): set up 2 sockets
Sep 29 14:47:55 dist-test-slave-5hd1 krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260929 14:47:56.316811 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:56 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:47:56.452068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:56.573031 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:56.573323 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:56.573366 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:56.577494 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:56.577560 7864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:47:56.577577 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:56.577595 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:56.577612 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:56.577670 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:56.583005 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:56.584105 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:56.585173 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:56.594393 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:56.594616 7864 server_base.cc:1034] running on GCE node
W20260929 14:47:56.594445 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:56.594448 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:56.595456 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:56.596662 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:56.597872 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790693276597832 us; error 60 us; skew 500 ppm
Sep 29 14:47:56 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:56.607137 7864 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:47:56.608160 7864 webserver.cc:460] Webserver started at http://127.0.98.62:41483/ using document root <none> and password file <none>
I20260929 14:47:56.608840 7864 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:56.608935 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:56.609171 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:56.611187 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "043409dd056040d29e35b87091f36a37"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "814f8f21a99d642b9dc9988169798e61"
server_key_iv: "e897b638713e8270cf0dc9c2eb8afbc8"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.611750 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "043409dd056040d29e35b87091f36a37"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "814f8f21a99d642b9dc9988169798e61"
server_key_iv: "e897b638713e8270cf0dc9c2eb8afbc8"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.615746 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 14:47:56.618326 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:56.619789 7864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:56.619951 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "043409dd056040d29e35b87091f36a37"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "814f8f21a99d642b9dc9988169798e61"
server_key_iv: "e897b638713e8270cf0dc9c2eb8afbc8"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.620076 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:56.638302 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:56.641417 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:56.641620 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:56.650254 7864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46509
I20260929 14:47:56.650274 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46509 every 8 connection(s)
I20260929 14:47:56.651180 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:47:56.654691 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:56.659337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7864
I20260929 14:47:56.659451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:47:56.659786 392 external_mini_cluster.cc:1428] Setting key ab65a50b83b74e01b7e3b2ab4353a44b
I20260929 14:47:56.662048 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Bootstrap starting.
I20260929 14:47:56.664906 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:56.665800 7942 log.cc:824] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:56.668344 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: No bootstrap required, opened a new log
Sep 29 14:47:56 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:56.673041 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260929 14:47:56.673357 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:56.673448 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043409dd056040d29e35b87091f36a37, State: Initialized, Role: FOLLOWER
I20260929 14:47:56.674067 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260929 14:47:56.674271 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:47:56.674350 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:47:56.674471 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:56.675719 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260929 14:47:56.675984 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:56.661491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.0.98.62:46509)
0929 14:47:56.662061 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:56.662079 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:56.662119 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:47:56.662999 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:56.663019 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:56.663387 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:56.663784 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:56.664609 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.665698 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.666585 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.667078 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.667254 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:56.670431 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:56.670436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:56.670454 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:56.670484 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:56.672608 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.673080 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.673086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.673091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.673207 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.673543 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.673546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.673548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.673983 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:56.674149 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:56.674711 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 14:47:56.675080 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":170,"threads_started":1}
I20260929 14:47:56.676231 7942 leader_election.cc:302] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043409dd056040d29e35b87091f36a37; no voters:
I20260929 14:47:56.676661 7942 leader_election.cc:288] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:47:56.676812 7947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:56.677091 7947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [term 1 LEADER]: Becoming Leader. State: Replica: 043409dd056040d29e35b87091f36a37, State: Running, Role: LEADER
I20260929 14:47:56.677533 7947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260929 14:47:56.678218 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:47:56.678813 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043409dd056040d29e35b87091f36a37. Latest consensus state: current_term: 1 leader_uuid: "043409dd056040d29e35b87091f36a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260929 14:47:56.678947 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:56.679246 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043409dd056040d29e35b87091f36a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043409dd056040d29e35b87091f36a37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260929 14:47:56.679333 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37 [sys.catalog]: This master's current role is: LEADER
I20260929 14:47:56.679826 7952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:47:56.683281 7952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:47:56.690116 7952 catalog_manager.cc:1347] Generated new cluster ID: 87c8dc9741104f35966027c658fd2688
I20260929 14:47:56.690229 7952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:47:56.708730 7952 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:47:56.709969 7952 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:47:56.717008 7952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Generated new TSK 0
I20260929 14:47:56.718003 7952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:47:56.759500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:56.892534 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:56.892802 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:56.892846 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:56.896862 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:56.896934 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:56.896973 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:56.897011 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:47:56.902125 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:56.903126 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:56.904086 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:56.914494 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:56.914497 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:56.914638 7968 server_base.cc:1034] running on GCE node
W20260929 14:47:56.914497 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:56.915453 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:56.916332 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:56.917528 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790693276917512 us; error 33 us; skew 500 ppm
Sep 29 14:47:56 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:56.920836 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:47:56.921963 7968 webserver.cc:460] Webserver started at http://127.0.98.1:44611/ using document root <none> and password file <none>
I20260929 14:47:56.922659 7968 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:56.922760 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:56.923041 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:56.925741 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e0c33b7fa6f48ca8047965a6ab719ca"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "7f09e5b31dabd864909451e6793239a9"
server_key_iv: "51a0f2369fdd0a2527c874618591fbca"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.926383 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e0c33b7fa6f48ca8047965a6ab719ca"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "7f09e5b31dabd864909451e6793239a9"
server_key_iv: "51a0f2369fdd0a2527c874618591fbca"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.931262 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 14:47:56.934501 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:56.936288 7968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:56.936460 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "9e0c33b7fa6f48ca8047965a6ab719ca"
format_stamp: "Formatted at 2026-09-29 14:47:56 on dist-test-slave-5hd1"
server_key: "7f09e5b31dabd864909451e6793239a9"
server_key_iv: "51a0f2369fdd0a2527c874618591fbca"
server_key_version: "encryptionkey@0"
I20260929 14:47:56.936621 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:56.955099 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:56.959060 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:56.959317 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:56.961264 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:56.961324 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:56.961367 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:56.961422 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:56 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:56.974387 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37527
I20260929 14:47:56.974484 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37527 every 8 connection(s)
I20260929 14:47:56.975497 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:47:56.977128 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:56.962476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48505 (local address 127.0.98.62:46509)
0929 14:47:56.962610 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:56.962615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:56.963975 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 14:47:56.965385 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:56.965388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:56.965443 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:56.965528 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:56.967298 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.968030 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.969473 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.969985 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.970179 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:56.973003 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:56.973007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:56.973010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:56.973044 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:56.974779 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.975297 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.975301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.975303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.975389 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.975776 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.975780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.975781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.976003 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:56.976102 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:56.976807 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 14:47:56.976973 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:47:56.977746 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:56.963203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.1:48505)
0929 14:47:56.963810 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:56.963861 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:56.965151 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:56.965801 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:56.965810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:56.966317 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:56.967086 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:56.967106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.968242 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.968245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:56.969223 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:56.969237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.970240 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.970245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:56.970423 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:56.971117 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:56.971138 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:56.972812 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:56.974901 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:56.974907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:56.974921 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:56.975187 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:56.975488 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:56.975491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:56.975493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:56.975678 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:56.976155 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:56.976161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:56.976488 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 14:47:56.976675 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 14:47:56.976926 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":167,"threads_started":1}
I20260929 14:47:56.978390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 7968
I20260929 14:47:56.978533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:47:56.978992 392 external_mini_cluster.cc:1428] Setting key 5523cf993781f24ebabe7bcc53181383
I20260929 14:47:56.979197 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260929 14:47:56.979393 8099 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:56.980280 8099 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260929 14:47:56.982689 7895 ts_manager.cc:194] Registered new tserver with Master: 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527)
I20260929 14:47:56.984548 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48505
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:47:56.992815 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:56.981901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36643 (local address 127.0.98.62:46509)
0929 14:47:56.982025 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:56.982029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:56.984485 (+ 2456us) server_negotiation.cc:407] Connection header received
0929 14:47:56.984679 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:56.984682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:56.984773 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:56.984830 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:56.985595 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.986364 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.987512 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.987797 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.987860 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:56.989353 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:56.989357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:56.989360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:56.989383 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:56.991070 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.991499 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.991505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.991510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.991596 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:56.991876 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:56.991881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:56.991883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:56.992086 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:56.992213 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:56.992480 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 14:47:56.992605 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:47:56.992820 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:56.981792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.1:36643)
0929 14:47:56.984393 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:56.984407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:56.984551 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:56.985052 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:56.985057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:56.985297 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:56.985450 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:56.985460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.986556 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.986560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:56.987354 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:56.987366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:56.988007 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:56.988010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:56.988088 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:56.988790 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:56.988804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:56.989205 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:56.991177 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:56.991180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:56.991183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:56.991398 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:56.991695 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:56.991698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:56.991700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:56.991789 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:56.992209 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:56.992212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:56.992317 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 14:47:56.992427 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 14:47:56.992612 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2515,"thread_start_us":173,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:47:57.012817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:57.124467 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:57.124871 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:57.124948 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:57.129272 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:57.129335 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:57.129407 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:57.129503 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:47:57.135030 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:57.136078 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:57.137221 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:57.147651 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:57.147740 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:57.147647 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:57.148005 8108 server_base.cc:1034] running on GCE node
I20260929 14:47:57.148661 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:57.149394 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:57.150595 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790693277150575 us; error 41 us; skew 500 ppm
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:57.153638 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:47:57.154593 8108 webserver.cc:460] Webserver started at http://127.0.98.2:46755/ using document root <none> and password file <none>
I20260929 14:47:57.155154 8108 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:57.155227 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:57.155402 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:57.157510 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "528ede259700460c97efaef3ecfaa156"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "5ffb1d29e0690488c0f234f97485eecc"
server_key_iv: "713b14fb5f9d1fcade04c83cd1899e13"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.158026 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "528ede259700460c97efaef3ecfaa156"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "5ffb1d29e0690488c0f234f97485eecc"
server_key_iv: "713b14fb5f9d1fcade04c83cd1899e13"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.162149 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 14:47:57.165153 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:57.166694 8108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:47:57.166862 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "528ede259700460c97efaef3ecfaa156"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "5ffb1d29e0690488c0f234f97485eecc"
server_key_iv: "713b14fb5f9d1fcade04c83cd1899e13"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.167011 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:57.183991 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:57.187136 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:57.187325 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:57.189007 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:57.189075 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:57.189131 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:57.189153 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:57.201340 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38213
I20260929 14:47:57.201535 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38213 every 8 connection(s)
I20260929 14:47:57.202353 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:47:57.206034 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.190003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38947 (local address 127.0.98.62:46509)
0929 14:47:57.190195 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.190203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.191366 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 14:47:57.192974 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.192978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.193046 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.193158 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.195124 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.196066 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.197576 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.197922 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.198044 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.201781 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.201786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.201789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.201823 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.203611 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.204275 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.204278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.204280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.204370 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.204738 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.204742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.204743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.204973 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.205084 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.205691 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.205831 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:47:57.207059 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.190583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.2:38947)
0929 14:47:57.191181 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.191239 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.192697 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.193419 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.193430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.194012 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:57.194915 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.194944 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.196234 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.196238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.197439 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.197452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.198120 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.198125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.198419 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.199320 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:57.199351 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:57.201582 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:57.203739 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.203746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.203765 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.204117 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.204479 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.204482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.204484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.204635 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.205092 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:57.205101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:57.205398 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.205639 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.206033 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":163,"threads_started":1}
I20260929 14:47:57.208518 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260929 14:47:57.208787 8239 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:57.209458 8239 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260929 14:47:57.210368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8108
I20260929 14:47:57.210503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:47:57.210789 7896 ts_manager.cc:194] Registered new tserver with Master: 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213)
I20260929 14:47:57.210822 392 external_mini_cluster.cc:1428] Setting key 75d13703ca432ea2ead81ed35eafc4e6
I20260929 14:47:57.211710 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38947
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:47:57.221901 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.212108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.2:41227)
0929 14:47:57.212514 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.212533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.212678 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.212991 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.212998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.213297 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:57.213497 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.213508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.214486 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.214490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.215134 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.215147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.215743 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.215749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.215880 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.216899 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:57.216918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:57.217514 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:57.219606 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.219613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.219617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.219939 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.220348 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.220353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.220357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.220493 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.221096 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:57.221102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:57.221259 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.221397 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.221595 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":179,"threads_started":1}
I20260929 14:47:57.221915 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.211350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41227 (local address 127.0.98.62:46509)
0929 14:47:57.211518 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.211522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.212642 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 14:47:57.212792 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.212795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.212846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.212940 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.213639 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.214335 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.215329 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.215598 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.215706 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.217713 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.217717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.217719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.217744 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.219464 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.220057 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.220059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.220061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.220140 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.220659 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.220664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.220666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.220955 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.221077 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.221506 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.221640 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:47:57.244930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:57.376792 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:57.377163 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:57.377247 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:57.382098 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:57.382187 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:57.382263 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:47:57.382344 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:47:57.387993 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:57.389348 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:57.390597 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:47:57.402263 8248 server_base.cc:1034] running on GCE node
W20260929 14:47:57.402266 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:57.402245 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:57.402251 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:57.403228 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:47:57.404026 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:47:57.405247 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790693277405223 us; error 45 us; skew 500 ppm
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:47:57.408627 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:47:57.409795 8248 webserver.cc:460] Webserver started at http://127.0.98.3:43261/ using document root <none> and password file <none>
I20260929 14:47:57.410493 8248 fs_manager.cc:359] Metadata directory not provided
I20260929 14:47:57.410561 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:47:57.410820 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:47:57.413362 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "8730db354102db3fc4fdb462dec349c8"
server_key_iv: "6f15db0689ac32ff9896e0168e40d7ef"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.414001 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "8730db354102db3fc4fdb462dec349c8"
server_key_iv: "6f15db0689ac32ff9896e0168e40d7ef"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.418679 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:47:57.421945 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:57.423691 8248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:57.423877 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
format_stamp: "Formatted at 2026-09-29 14:47:57 on dist-test-slave-5hd1"
server_key: "8730db354102db3fc4fdb462dec349c8"
server_key_iv: "6f15db0689ac32ff9896e0168e40d7ef"
server_key_version: "encryptionkey@0"
I20260929 14:47:57.424072 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:47:57.441661 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:47:57.445760 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:47:57.446014 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:47:57.447877 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:47:57.447934 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:47:57.447983 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:47:57.448045 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:57.461285 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40035
I20260929 14:47:57.461469 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40035 every 8 connection(s)
I20260929 14:47:57.462733 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:47:57.463353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8248
I20260929 14:47:57.463500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:47:57.463886 392 external_mini_cluster.cc:1428] Setting key ad1af11f6b28f115eed79e48f4e963e2
I20260929 14:47:57.465348 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.449380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58819 (local address 127.0.98.62:46509)
0929 14:47:57.449532 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.449536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.450925 (+ 1389us) server_negotiation.cc:407] Connection header received
0929 14:47:57.452599 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.452603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.452680 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.452780 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.454591 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.455482 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.457402 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.457765 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.457879 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.460734 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.460738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.460741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.460798 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.462737 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.463555 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.463561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.463563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.463672 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.464067 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.464070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.464072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.464354 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.464473 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.465026 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.465166 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:47:57.466408 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.450017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.3:58819)
0929 14:47:57.450711 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.450785 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.452279 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.453028 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.453040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.453647 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:57.454413 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.454452 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.455618 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.455622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.457233 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.457247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.457925 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.457929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.458132 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.458842 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:57.458866 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:57.460540 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:57.462954 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.462961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.462995 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.463412 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.463784 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.463787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.463790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.463953 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.464485 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:57.464494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:57.464774 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.464973 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.465353 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":161,"threads_started":1}
I20260929 14:47:57.468029 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260929 14:47:57.468278 8379 heartbeater.cc:464] Registering TS with master...
I20260929 14:47:57.468959 8379 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260929 14:47:57.470242 7896 ts_manager.cc:194] Registered new tserver with Master: 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:57.471066 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58819
I20260929 14:47:57.478291 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:47:57.478890 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.470745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40427 (local address 127.0.98.62:46509)
0929 14:47:57.470896 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.470901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.471116 (+ 215us) server_negotiation.cc:407] Connection header received
0929 14:47:57.471287 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.471290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.471340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.471443 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.472159 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.472777 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.473778 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.473986 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.474068 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.475213 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.475216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.475218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.475239 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.476893 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.477402 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.477408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.477410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.477506 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.477861 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.477864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.477866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.478086 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.478215 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.478636 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.478764 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:47:57.478947 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.470618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.3:40427)
0929 14:47:57.471016 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.471029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.471185 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.471615 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.471619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.471905 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:57.472022 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.472030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.472921 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.472924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.473662 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.473671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.474097 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.474100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.474172 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.474733 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:57.474744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:57.475083 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:57.477012 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.477015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.477017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.477292 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.477654 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.477657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.477659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.477760 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.478217 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:57.478221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:57.478346 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.478543 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.478727 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":165,"threads_started":1}
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:47:57.504359 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:57.497959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.0.98.62:46509)
0929 14:47:57.498119 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.498123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.498135 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:47:57.498174 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.498177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.498216 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.498318 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.499047 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.499580 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.500497 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.500708 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.500824 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.504225 (+ 3401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38344: BlockingRecv error: recv got EOF from 127.0.0.1:38344 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:57.515246 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.506367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.98.62:46509)
0929 14:47:57.506573 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.506578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.506615 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:47:57.506756 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.506759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.506810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.506887 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.507597 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.508136 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.509069 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.509304 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.509431 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.511870 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.511873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.511876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.511902 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.513496 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.513959 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.513962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.513964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.514044 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.514325 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.514327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.514328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.514520 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.514641 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.514848 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.514993 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 14:47:57.519068 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:57.522465 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:57.540431 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.533515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40035 (local address 127.0.0.1:44570)
0929 14:47:57.534535 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.534599 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.534771 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.535428 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.535435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.535467 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:57.535660 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.535671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537375 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.537379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.538684 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.538693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.539739 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.539743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.539868 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.539921 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.540023 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.540205 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":125,"spinlock_wait_cycles":4480,"thread_start_us":139,"threads_started":1}
I20260929 14:47:57.540963 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.533682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.0.98.3:40035)
0929 14:47:57.534079 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.534091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.534810 (+ 719us) server_negotiation.cc:407] Connection header received
0929 14:47:57.534938 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.534946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.535207 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.535835 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:57.535978 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537233 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.538839 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.539604 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.540381 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.540613 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.540680 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.540800 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260929 14:47:57.541430 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.533639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37527 (local address 127.0.0.1:52284)
0929 14:47:57.534535 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.534599 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.534771 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.535717 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.535722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.535744 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:57.535837 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.535842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537386 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.537391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.539323 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.539338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.540758 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.540762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.540963 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.541039 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.541110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.541225 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":300,"spinlock_wait_cycles":8832,"thread_start_us":102,"threads_started":1}
I20260929 14:47:57.541618 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.534922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38213 (local address 127.0.0.1:49430)
0929 14:47:57.535214 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.535228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.535319 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.536262 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.536265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.536285 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:57.536423 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.536429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537893 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.537897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.539323 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.539338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.540758 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.540762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.540958 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.541041 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.541110 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.541481 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":518,"thread_start_us":119,"threads_started":1}
I20260929 14:47:57.541810 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.535375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.0.98.2:38213)
0929 14:47:57.535778 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.535791 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.535813 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:47:57.535905 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.535909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.536075 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.536563 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:57.536731 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537726 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.539515 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.540612 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.541171 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.541381 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.541442 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.541558 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260929 14:47:57.542124 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.534252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.0.98.1:37527)
0929 14:47:57.535141 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.535150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.535179 (+ 29us) server_negotiation.cc:407] Connection header received
0929 14:47:57.535277 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.535284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.535497 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.535732 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:57.536008 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.537234 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.539500 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.540597 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.541689 (+ 1092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.541841 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.541899 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.542001 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":90,"threads_started":1}
I20260929 14:47:57.545068 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 (DEFAULT_TABLE table=test-table [id=d774405e33fe492f884785ab2a7e0f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.545078 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 (DEFAULT_TABLE table=test-table [id=d774405e33fe492f884785ab2a7e0f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.545056 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 (DEFAULT_TABLE table=test-table [id=d774405e33fe492f884785ab2a7e0f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.546330 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470bb187f34f45ce9dde1d62f17607e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.546339 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470bb187f34f45ce9dde1d62f17607e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.546339 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470bb187f34f45ce9dde1d62f17607e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.555653 8407 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Bootstrap starting.
I20260929 14:47:57.556247 8409 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Bootstrap starting.
I20260929 14:47:57.556571 8408 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Bootstrap starting.
I20260929 14:47:57.558588 8407 tablet_bootstrap.cc:654] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.558579 8409 tablet_bootstrap.cc:654] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.558604 8408 tablet_bootstrap.cc:654] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.559391 8408 log.cc:824] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:57.559579 8409 log.cc:824] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:57.559588 8407 log.cc:824] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:47:57.561766 8408 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: No bootstrap required, opened a new log
I20260929 14:47:57.561766 8407 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: No bootstrap required, opened a new log
I20260929 14:47:57.562009 8408 ts_tablet_manager.cc:1397] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:47:57.562136 8407 ts_tablet_manager.cc:1397] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:47:57.562748 8409 tablet_bootstrap.cc:492] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: No bootstrap required, opened a new log
I20260929 14:47:57.562942 8409 ts_tablet_manager.cc:1397] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:47:57.565549 8407 raft_consensus.cc:348] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.565824 8407 raft_consensus.cc:374] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.565876 8407 raft_consensus.cc:729] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e0c8f926a64ad6a6ab9daabccb36f6, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.566326 8407 consensus_queue.cc:260] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.567034 8379 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260929 14:47:57.567199 8409 raft_consensus.cc:348] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.567200 8408 raft_consensus.cc:348] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.567497 8408 raft_consensus.cc:374] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.567509 8409 raft_consensus.cc:374] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.567565 8409 raft_consensus.cc:729] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.567574 8408 raft_consensus.cc:729] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.568150 8408 consensus_queue.cc:260] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.568150 8409 consensus_queue.cc:260] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.568605 8407 ts_tablet_manager.cc:1428] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:47:57.569303 8408 ts_tablet_manager.cc:1428] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20260929 14:47:57.569334 8409 ts_tablet_manager.cc:1428] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 14:47:57.569728 8099 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260929 14:47:57.569749 8239 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
W20260929 14:47:57.704596 8240 tablet.cc:2369] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:57.704757 8415 raft_consensus.cc:482] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:57.705108 8415 raft_consensus.cc:504] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.706612 8415 leader_election.cc:288] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035), 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213)
I20260929 14:47:57.711537 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.707145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52193 (local address 127.0.98.3:40035)
0929 14:47:57.707325 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.707331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.707758 (+ 427us) server_negotiation.cc:407] Connection header received
0929 14:47:57.707930 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.707932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.708011 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.708099 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:57.708561 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.709276 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.710475 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.710901 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.710988 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.711092 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.711298 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.711394 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:47:57.711566 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.707029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40035 (local address 127.0.98.1:52193)
0929 14:47:57.707610 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.707629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.707809 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.708139 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.708143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.708165 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:57.708362 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.708367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.709396 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.709398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.710340 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.710349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.711042 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.711046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.711143 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.711174 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.711259 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.711360 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":3,"thread_start_us":147,"threads_started":1}
I20260929 14:47:57.711675 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.707137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37889 (local address 127.0.98.2:38213)
0929 14:47:57.707279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.707316 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.707606 (+ 290us) server_negotiation.cc:407] Connection header received
0929 14:47:57.707836 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.707843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.707920 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.708004 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:57.708439 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.709103 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.710527 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.710934 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.711067 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.711141 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.711434 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.711552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:47:57.712103 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.707003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38213 (local address 127.0.98.1:37889)
0929 14:47:57.707488 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.707502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.707669 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.708052 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.708057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.708080 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:57.708317 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.708325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.709214 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.709218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.710369 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.710378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.711093 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.711097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.711253 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.711295 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.711854 (+ 559us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.711963 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":28,"thread_start_us":196,"threads_started":1}
I20260929 14:47:57.712551 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470bb187f34f45ce9dde1d62f17607e1" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" is_pre_election: true
I20260929 14:47:57.712857 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470bb187f34f45ce9dde1d62f17607e1" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156" is_pre_election: true
I20260929 14:47:57.712940 8333 raft_consensus.cc:2454] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.713178 8193 raft_consensus.cc:2454] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.713539 7985 leader_election.cc:302] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e0c8f926a64ad6a6ab9daabccb36f6, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.713869 8415 raft_consensus.cc:2781] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:57.713958 8415 raft_consensus.cc:482] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:57.713989 8415 raft_consensus.cc:3037] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:47:57.714208 8380 tablet.cc:2369] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:57.715157 8415 raft_consensus.cc:504] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.715560 8415 leader_election.cc:288] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Requested vote from peers 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035), 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213)
I20260929 14:47:57.715955 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470bb187f34f45ce9dde1d62f17607e1" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156"
I20260929 14:47:57.716068 8193 raft_consensus.cc:3037] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.716073 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470bb187f34f45ce9dde1d62f17607e1" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
I20260929 14:47:57.716252 8333 raft_consensus.cc:3037] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.717176 8193 raft_consensus.cc:2454] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.717561 7987 leader_election.cc:302] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.717828 8415 raft_consensus.cc:2781] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:57.718109 8415 raft_consensus.cc:686] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Becoming Leader. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Running, Role: LEADER
I20260929 14:47:57.718164 8333 raft_consensus.cc:2454] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.718534 8415 consensus_queue.cc:237] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } }
I20260929 14:47:57.722391 7895 catalog_manager.cc:5495] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } health_report { overall_health: UNKNOWN } } }
W20260929 14:47:57.727121 8100 tablet.cc:2369] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:47:57.779604 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.775497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.0.98.1:37527)
0929 14:47:57.775633 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.775637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.775655 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:47:57.775814 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.775816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.775879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.776000 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:57.776300 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.776803 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.777787 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.777999 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.778084 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.778491 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:57.779155 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:57.779269 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.779405 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.779485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:47:57.786406 8193 raft_consensus.cc:1261] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:57.786566 8333 raft_consensus.cc:1261] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:57.787199 8418 consensus_queue.cc:1035] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:57.787382 8415 consensus_queue.cc:1035] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:57.808367 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:57.808533 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:57.811271 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Sending DeleteTablet for 3 replicas of tablet 470bb187f34f45ce9dde1d62f17607e1
I20260929 14:47:57.812386 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52284
I20260929 14:47:57.812386 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44570
I20260929 14:47:57.812893 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 470bb187f34f45ce9dde1d62f17607e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49430
I20260929 14:47:57.813473 8432 tablet_replica.cc:330] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: stopping tablet replica
I20260929 14:47:57.813520 8430 tablet_replica.cc:330] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: stopping tablet replica
I20260929 14:47:57.813583 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:47:57.813913 8430 raft_consensus.cc:2229] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Raft consensus shutting down.
W20260929 14:47:57.814007 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:57.814237 8432 raft_consensus.cc:2229] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:57.814414 8430 raft_consensus.cc:2258] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:57.814863 8431 tablet_replica.cc:330] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: stopping tablet replica
I20260929 14:47:57.815160 8431 raft_consensus.cc:2229] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:57.815492 8431 raft_consensus.cc:2258] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:57.815773 8432 raft_consensus.cc:2258] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:57.816383 8430 ts_tablet_manager.cc:1905] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:57.817795 8431 ts_tablet_manager.cc:1905] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:57.819557 8430 ts_tablet_manager.cc:1918] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:57.819747 8430 log.cc:1196] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/470bb187f34f45ce9dde1d62f17607e1
I20260929 14:47:57.820195 8430 ts_tablet_manager.cc:1939] T 470bb187f34f45ce9dde1d62f17607e1 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting consensus metadata
I20260929 14:47:57.820109 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 32adc9d2f03d44db9731732253edbe84 (DEFAULT_TABLE table=test-table [id=56ff775f92ed443cbafd04d7e615c6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.820448 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32adc9d2f03d44db9731732253edbe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.821395 8431 ts_tablet_manager.cc:1918] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:57.821514 8431 log.cc:1196] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/470bb187f34f45ce9dde1d62f17607e1
I20260929 14:47:57.821548 8432 ts_tablet_manager.cc:1905] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:57.821848 8431 ts_tablet_manager.cc:1939] T 470bb187f34f45ce9dde1d62f17607e1 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting consensus metadata
I20260929 14:47:57.822942 8407 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Bootstrap starting.
I20260929 14:47:57.823323 7881 catalog_manager.cc:4841] TS 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035): tablet 470bb187f34f45ce9dde1d62f17607e1 (table test-table [id=d774405e33fe492f884785ab2a7e0f1d]) successfully deleted
I20260929 14:47:57.823771 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 32adc9d2f03d44db9731732253edbe84 (DEFAULT_TABLE table=test-table [id=56ff775f92ed443cbafd04d7e615c6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.824028 8407 tablet_bootstrap.cc:654] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.824095 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32adc9d2f03d44db9731732253edbe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.824497 8432 ts_tablet_manager.cc:1918] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:57.824591 8432 log.cc:1196] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/470bb187f34f45ce9dde1d62f17607e1
I20260929 14:47:57.824904 8432 ts_tablet_manager.cc:1939] T 470bb187f34f45ce9dde1d62f17607e1 P 528ede259700460c97efaef3ecfaa156: Deleting consensus metadata
I20260929 14:47:57.825831 8407 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: No bootstrap required, opened a new log
I20260929 14:47:57.825945 8407 ts_tablet_manager.cc:1397] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:47:57.826503 7883 catalog_manager.cc:4841] TS 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213): tablet 470bb187f34f45ce9dde1d62f17607e1 (table test-table [id=d774405e33fe492f884785ab2a7e0f1d]) successfully deleted
I20260929 14:47:57.826471 8407 raft_consensus.cc:348] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.826609 8407 raft_consensus.cc:374] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.826653 8407 raft_consensus.cc:729] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e0c8f926a64ad6a6ab9daabccb36f6, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.826845 8407 consensus_queue.cc:260] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.827069 8409 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Bootstrap starting.
I20260929 14:47:57.827801 8407 ts_tablet_manager.cc:1428] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:57.828054 8409 tablet_bootstrap.cc:654] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.828960 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 32adc9d2f03d44db9731732253edbe84 (DEFAULT_TABLE table=test-table [id=56ff775f92ed443cbafd04d7e615c6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:57.829070 7882 catalog_manager.cc:4841] TS 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527): tablet 470bb187f34f45ce9dde1d62f17607e1 (table test-table [id=d774405e33fe492f884785ab2a7e0f1d]) successfully deleted
I20260929 14:47:57.829267 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32adc9d2f03d44db9731732253edbe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.830958 8409 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: No bootstrap required, opened a new log
I20260929 14:47:57.831055 8409 ts_tablet_manager.cc:1397] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 14:47:57.831523 8409 raft_consensus.cc:348] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.831650 8409 raft_consensus.cc:374] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.831681 8409 raft_consensus.cc:729] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.831929 8408 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Bootstrap starting.
I20260929 14:47:57.831956 8409 consensus_queue.cc:260] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.832942 8408 tablet_bootstrap.cc:654] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.832957 8409 ts_tablet_manager.cc:1428] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:57.834151 8408 tablet_bootstrap.cc:492] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: No bootstrap required, opened a new log
I20260929 14:47:57.834242 8408 ts_tablet_manager.cc:1397] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:57.834744 8408 raft_consensus.cc:348] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.834867 8408 raft_consensus.cc:374] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.834908 8408 raft_consensus.cc:729] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.835028 8408 consensus_queue.cc:260] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.835928 8408 ts_tablet_manager.cc:1428] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:57.858793 8415 raft_consensus.cc:482] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:57.858971 8415 raft_consensus.cc:504] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.859428 8415 leader_election.cc:288] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:57.859869 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32adc9d2f03d44db9731732253edbe84" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" is_pre_election: true
I20260929 14:47:57.859974 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32adc9d2f03d44db9731732253edbe84" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156" is_pre_election: true
I20260929 14:47:57.860044 8333 raft_consensus.cc:2454] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.860123 8193 raft_consensus.cc:2454] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.860488 7987 leader_election.cc:302] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.860719 8415 raft_consensus.cc:2781] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:57.860829 8415 raft_consensus.cc:482] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:57.860872 8415 raft_consensus.cc:3037] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.862354 8415 raft_consensus.cc:504] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.862784 8415 leader_election.cc:288] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Requested vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:57.863168 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32adc9d2f03d44db9731732253edbe84" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156"
I20260929 14:47:57.863194 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32adc9d2f03d44db9731732253edbe84" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
I20260929 14:47:57.863281 8193 raft_consensus.cc:3037] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.863328 8333 raft_consensus.cc:3037] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.864316 8193 raft_consensus.cc:2454] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.864598 7987 leader_election.cc:302] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.864621 8333 raft_consensus.cc:2454] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.864799 8415 raft_consensus.cc:2781] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:57.865001 8415 raft_consensus.cc:686] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Becoming Leader. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Running, Role: LEADER
I20260929 14:47:57.865156 8415 consensus_queue.cc:237] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.867020 7896 catalog_manager.cc:5495] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } }
Sep 29 14:47:57 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:57.900044 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.889929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.0.1:38368)
0929 14:47:57.890231 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.890244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.890346 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.890732 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.890735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.891137 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:47:57.891253 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.891259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.892489 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.892492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.893073 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.893082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.893841 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.893844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.893915 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.894642 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:47:57.894654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:47:57.896153 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:47:57.898308 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.898316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.898321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.898597 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.898909 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:47:57.898911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:47:57.898913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:47:57.898997 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:47:57.899479 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:47:57.899485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:47:57.899611 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.899728 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.899899 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260929 14:47:57.900233 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.890066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.0.98.62:46509)
0929 14:47:57.890240 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.890245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.890327 (+ 82us) server_negotiation.cc:407] Connection header received
0929 14:47:57.890466 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.890470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.890533 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.890663 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:57.891408 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.892327 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.893322 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.893678 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.893799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.896379 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:57.896383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:57.896386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:57.896414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:57.898177 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.898703 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.898707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.898709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.898797 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:57.899093 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:57.899096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:57.899098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:57.899327 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:57.899432 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.899803 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.899989 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":76}
I20260929 14:47:57.904745 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38368:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:47:57.905483 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:57.910794 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 55a8963b8f8243cf812fafaddcfbdf98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eba7f76f71024248981d5469270cd2f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:57.910943 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 55a8963b8f8243cf812fafaddcfbdf98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eba7f76f71024248981d5469270cd2f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:57.910943 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 55a8963b8f8243cf812fafaddcfbdf98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eba7f76f71024248981d5469270cd2f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:47:57.911325 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a8963b8f8243cf812fafaddcfbdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.911350 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a8963b8f8243cf812fafaddcfbdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.911350 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a8963b8f8243cf812fafaddcfbdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:57.914417 8407 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: Bootstrap starting.
I20260929 14:47:57.914604 8409 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: Bootstrap starting.
I20260929 14:47:57.915462 8409 tablet_bootstrap.cc:654] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.915601 8407 tablet_bootstrap.cc:654] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.916975 8408 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: Bootstrap starting.
I20260929 14:47:57.916999 8407 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: No bootstrap required, opened a new log
I20260929 14:47:57.917085 8407 ts_tablet_manager.cc:1397] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 14:47:57.917596 8407 raft_consensus.cc:348] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.917742 8407 raft_consensus.cc:374] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.917775 8407 raft_consensus.cc:729] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e0c8f926a64ad6a6ab9daabccb36f6, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.917773 8408 tablet_bootstrap.cc:654] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:57.917934 8407 consensus_queue.cc:260] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.918255 8407 ts_tablet_manager.cc:1428] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:47:57.918495 8413 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.918663 8413 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.919184 8409 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: No bootstrap required, opened a new log
I20260929 14:47:57.919301 8409 ts_tablet_manager.cc:1397] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 14:47:57.919826 8409 raft_consensus.cc:348] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.919968 8409 raft_consensus.cc:374] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.919971 8408 tablet_bootstrap.cc:492] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: No bootstrap required, opened a new log
I20260929 14:47:57.920001 8409 raft_consensus.cc:729] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.920060 8408 ts_tablet_manager.cc:1397] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:57.920470 8409 consensus_queue.cc:260] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.920578 8408 raft_consensus.cc:348] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.920698 8408 raft_consensus.cc:374] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:57.920728 8408 raft_consensus.cc:729] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Initialized, Role: FOLLOWER
I20260929 14:47:57.920753 8414 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.920897 8414 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.920862 8408 consensus_queue.cc:260] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.921149 8408 ts_tablet_manager.cc:1428] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:57.921769 8409 ts_tablet_manager.cc:1428] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:57.921933 8415 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.922080 8415 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.949038 8415 raft_consensus.cc:482] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:57.949236 8415 raft_consensus.cc:504] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.949796 8415 leader_election.cc:288] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:57.950178 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a8963b8f8243cf812fafaddcfbdf98" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156" is_pre_election: true
I20260929 14:47:57.950373 8193 raft_consensus.cc:2454] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.950476 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a8963b8f8243cf812fafaddcfbdf98" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" is_pre_election: true
I20260929 14:47:57.950663 8333 raft_consensus.cc:2454] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:57.950671 7987 leader_election.cc:302] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.950908 8415 raft_consensus.cc:2781] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:57.950989 8415 raft_consensus.cc:482] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:57.951046 8415 raft_consensus.cc:3037] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.952133 8415 raft_consensus.cc:504] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.952589 8415 leader_election.cc:288] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Requested vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:57.952914 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a8963b8f8243cf812fafaddcfbdf98" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156"
I20260929 14:47:57.953092 8193 raft_consensus.cc:3037] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.953035 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a8963b8f8243cf812fafaddcfbdf98" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
I20260929 14:47:57.953194 8333 raft_consensus.cc:3037] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:57.954077 8333 raft_consensus.cc:2454] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.954366 7985 leader_election.cc:302] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e0c8f926a64ad6a6ab9daabccb36f6, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:57.954535 8415 raft_consensus.cc:2781] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:57.954553 8193 raft_consensus.cc:2454] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:57.954625 8415 raft_consensus.cc:686] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Becoming Leader. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Running, Role: LEADER
I20260929 14:47:57.954771 8415 consensus_queue.cc:237] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:57.955534 8418 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: TxnStatusTablet state changed. Reason: New leader 9e0c33b7fa6f48ca8047965a6ab719ca. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.955647 8418 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:57.956066 8447 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:47:57.956272 8447 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:47:57.957149 7895 catalog_manager.cc:5495] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:57.979599 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.975925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.98.1:37527)
0929 14:47:57.976047 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.976051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.976154 (+ 103us) server_negotiation.cc:407] Connection header received
0929 14:47:57.976401 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.976404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.976459 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.976540 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:57.976790 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.977302 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.978351 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.978565 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.978670 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.979069 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:57.979138 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:57.979217 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:57.979402 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 14:47:57.979453 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:47:57.979599 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.975819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37527 (local address 127.0.0.1:52308)
0929 14:47:57.976068 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.976083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.976269 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.976570 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.976573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.976581 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:57.976691 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.976696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.977411 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.977416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.978216 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.978226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:57.978680 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:57.978684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:57.978855 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:57.978910 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:57.979222 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:57.979240 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:47:57.979332 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:47:57.979472 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 14:47:57.982328 8193 raft_consensus.cc:1261] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:57.982470 8333 raft_consensus.cc:1261] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:57.982892 8415 consensus_queue.cc:1035] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:57.983130 8418 consensus_queue.cc:1035] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:57.984469 8414 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: TxnStatusTablet state changed. Reason: New leader 9e0c33b7fa6f48ca8047965a6ab719ca. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.984582 8414 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.984855 8413 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: TxnStatusTablet state changed. Reason: New leader 9e0c33b7fa6f48ca8047965a6ab719ca. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.985038 8413 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.987016 8418 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.987157 8418 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:57.987970 8414 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.988207 8414 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 528ede259700460c97efaef3ecfaa156: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:57.988524 8421 mvcc.cc:205] Tried to move back new op lower bound from 7334679666610880512 to 7334679666504163328. Current Snapshot: MvccSnapshot[applied={T|T < 7334679666610880512 or (T in {7334679666610880512})}]
I20260929 14:47:57.989322 8418 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.989457 8418 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 9e0c33b7fa6f48ca8047965a6ab719ca: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:47:57.992940 8413 tablet_replica.cc:439] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } } }
I20260929 14:47:57.993108 8413 tablet_replica.cc:442] T 55a8963b8f8243cf812fafaddcfbdf98 P 67e0c8f926a64ad6a6ab9daabccb36f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:47:58.002177 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.998759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37527 (local address 127.0.98.1:46687)
0929 14:47:57.999044 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:57.999061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:57.999183 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:57.999405 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:57.999407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:57.999413 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:47:57.999563 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:57.999569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.000384 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.000391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.000999 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:58.001006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.001443 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.001446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.001521 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.001566 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:47:58.001854 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:47:58.001862 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:47:58.001926 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:47:58.002036 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":50,"thread_start_us":108,"threads_started":1}
I20260929 14:47:58.002175 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:57.998765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46687 (local address 127.0.98.1:37527)
0929 14:47:57.998895 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:57.998900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:57.999119 (+ 219us) server_negotiation.cc:407] Connection header received
0929 14:47:57.999269 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:57.999272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:57.999322 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:57.999399 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:57.999670 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.000282 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.001117 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.001334 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.001433 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.001694 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:58.001759 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:58.001853 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.001932 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.002014 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":43}
I20260929 14:47:58.008980 8333 raft_consensus.cc:1261] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.008980 8193 raft_consensus.cc:1261] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.009416 8451 consensus_queue.cc:1035] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.009577 8415 consensus_queue.cc:1035] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.014546 8422 mvcc.cc:205] Tried to move back new op lower bound from 7334679666720464896 to 7334679666137288704. Current Snapshot: MvccSnapshot[applied={T|T < 7334679666720464896}]
I20260929 14:47:58.048205 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.048393 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.050755 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Sending DeleteTablet for 3 replicas of tablet 32adc9d2f03d44db9731732253edbe84
I20260929 14:47:58.051537 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 32adc9d2f03d44db9731732253edbe84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49430
I20260929 14:47:58.051744 8432 tablet_replica.cc:330] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: stopping tablet replica
I20260929 14:47:58.051702 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 32adc9d2f03d44db9731732253edbe84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52284
I20260929 14:47:58.051851 8432 raft_consensus.cc:2229] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.051961 8432 raft_consensus.cc:2258] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.051937 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 32adc9d2f03d44db9731732253edbe84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44570
I20260929 14:47:58.052148 8431 tablet_replica.cc:330] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: stopping tablet replica
I20260929 14:47:58.052148 8430 tablet_replica.cc:330] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: stopping tablet replica
I20260929 14:47:58.052306 8431 raft_consensus.cc:2229] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.052320 8430 raft_consensus.cc:2229] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:58.052448 8431 raft_consensus.cc:2258] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.052538 8430 raft_consensus.cc:2258] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.053679 8430 ts_tablet_manager.cc:1905] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.053676 8432 ts_tablet_manager.cc:1905] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.056579 8430 ts_tablet_manager.cc:1918] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:58.056665 8430 log.cc:1196] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/32adc9d2f03d44db9731732253edbe84
I20260929 14:47:58.056658 8432 ts_tablet_manager.cc:1918] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:58.056748 8432 log.cc:1196] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/32adc9d2f03d44db9731732253edbe84
I20260929 14:47:58.056962 8430 ts_tablet_manager.cc:1939] T 32adc9d2f03d44db9731732253edbe84 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting consensus metadata
I20260929 14:47:58.057040 8432 ts_tablet_manager.cc:1939] T 32adc9d2f03d44db9731732253edbe84 P 528ede259700460c97efaef3ecfaa156: Deleting consensus metadata
I20260929 14:47:58.057258 8431 ts_tablet_manager.cc:1905] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.057936 7882 catalog_manager.cc:4841] TS 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527): tablet 32adc9d2f03d44db9731732253edbe84 (table test-table [id=56ff775f92ed443cbafd04d7e615c6bf]) successfully deleted
I20260929 14:47:58.058068 7883 catalog_manager.cc:4841] TS 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213): tablet 32adc9d2f03d44db9731732253edbe84 (table test-table [id=56ff775f92ed443cbafd04d7e615c6bf]) successfully deleted
I20260929 14:47:58.060590 8431 ts_tablet_manager.cc:1918] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:58.060688 8431 log.cc:1196] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/32adc9d2f03d44db9731732253edbe84
I20260929 14:47:58.060963 8431 ts_tablet_manager.cc:1939] T 32adc9d2f03d44db9731732253edbe84 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting consensus metadata
I20260929 14:47:58.061800 7881 catalog_manager.cc:4841] TS 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035): tablet 32adc9d2f03d44db9731732253edbe84 (table test-table [id=56ff775f92ed443cbafd04d7e615c6bf]) successfully deleted
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:47:58.090665 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:47:58.082922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.0.98.62:46509)
0929 14:47:58.083097 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.083102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.083405 (+ 303us) server_negotiation.cc:407] Connection header received
0929 14:47:58.083659 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.083662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.083733 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.083842 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:58.084783 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.085606 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.086544 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.086879 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.086988 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.090521 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38378: BlockingRecv error: recv got EOF from 127.0.0.1:38378 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:47:58.103096 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.092702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.98.62:46509)
0929 14:47:58.092864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.092872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.093075 (+ 203us) server_negotiation.cc:407] Connection header received
0929 14:47:58.093178 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.093181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.093232 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.093312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:47:58.094137 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.094684 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.095868 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.096134 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.096341 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.099380 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:47:58.099384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:47:58.099386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:47:58.099420 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:47:58.101226 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:58.101739 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:58.101748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:58.101750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:58.101828 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:47:58.102120 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:47:58.102125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:47:58.102127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:47:58.102340 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:47:58.102503 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.102746 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.102881 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 14:47:58.106472 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:47:58.107110 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:58.113026 8032 tablet_service.cc:1467] Processing CreateTablet for tablet e938fa1e74c1467882333233e774f815 (DEFAULT_TABLE table=test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.113395 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e938fa1e74c1467882333233e774f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.113394 8309 tablet_service.cc:1467] Processing CreateTablet for tablet e938fa1e74c1467882333233e774f815 (DEFAULT_TABLE table=test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.113778 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e938fa1e74c1467882333233e774f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.116052 8407 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Bootstrap starting.
I20260929 14:47:58.116307 8173 tablet_service.cc:1467] Processing CreateTablet for tablet e938fa1e74c1467882333233e774f815 (DEFAULT_TABLE table=test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.116588 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e938fa1e74c1467882333233e774f815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.117045 8407 tablet_bootstrap.cc:654] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.118203 8407 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: No bootstrap required, opened a new log
I20260929 14:47:58.118317 8407 ts_tablet_manager.cc:1397] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:47:58.118764 8407 raft_consensus.cc:348] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.118881 8409 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Bootstrap starting.
I20260929 14:47:58.118932 8407 raft_consensus.cc:374] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.118971 8407 raft_consensus.cc:729] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e0c8f926a64ad6a6ab9daabccb36f6, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.119107 8407 consensus_queue.cc:260] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.119462 8407 ts_tablet_manager.cc:1428] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:58.119822 8409 tablet_bootstrap.cc:654] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.121472 8409 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: No bootstrap required, opened a new log
I20260929 14:47:58.121564 8409 ts_tablet_manager.cc:1397] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 14:47:58.122009 8409 raft_consensus.cc:348] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.122146 8409 raft_consensus.cc:374] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.122181 8409 raft_consensus.cc:729] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.122330 8409 consensus_queue.cc:260] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.122609 8408 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Bootstrap starting.
I20260929 14:47:58.123482 8408 tablet_bootstrap.cc:654] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.124871 8408 tablet_bootstrap.cc:492] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: No bootstrap required, opened a new log
I20260929 14:47:58.124969 8408 ts_tablet_manager.cc:1397] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:58.125344 8408 raft_consensus.cc:348] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.125463 8408 raft_consensus.cc:374] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.125492 8408 raft_consensus.cc:729] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.125605 8408 consensus_queue.cc:260] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.125875 8408 ts_tablet_manager.cc:1428] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:58.126852 8409 ts_tablet_manager.cc:1428] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260929 14:47:58.233278 8414 raft_consensus.cc:482] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:58.233536 8414 raft_consensus.cc:504] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.234972 8414 leader_election.cc:288] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:58.240849 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.235400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32785 (local address 127.0.98.1:37527)
0929 14:47:58.235545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.235550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.235840 (+ 290us) server_negotiation.cc:407] Connection header received
0929 14:47:58.236033 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.236036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.236099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.236224 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:58.236681 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.237526 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.239400 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.240044 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.240204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.240304 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.240618 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.240697 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:47:58.240900 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.235546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42303 (local address 127.0.98.3:40035)
0929 14:47:58.235828 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.235833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.235852 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:47:58.236033 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.236036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.236099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.236224 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:47:58.236681 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.237531 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.239400 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.240125 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.240281 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.240348 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.240591 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.240690 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260929 14:47:58.240833 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.235327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37527 (local address 127.0.98.2:32785)
0929 14:47:58.235704 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:58.235720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:58.235887 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:58.236292 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:58.236296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:58.236336 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:58.236546 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:58.236552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.237657 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.237661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.239243 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:58.239255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.240258 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.240262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.240358 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.240448 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:47:58.240594 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 14:47:58.240642 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":244,"spinlock_wait_cycles":22272,"thread_start_us":96,"threads_started":1}
I20260929 14:47:58.240837 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.235400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40035 (local address 127.0.98.2:42303)
0929 14:47:58.235774 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:58.235786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:47:58.235887 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:47:58.236292 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:47:58.236303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:47:58.236335 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:47:58.236539 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:58.236548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.237657 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.237661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.239243 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:47:58.239255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.240298 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.240301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:47:58.240405 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.240446 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 14:47:58.240533 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:47:58.240624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":197,"spinlock_wait_cycles":44544,"thread_start_us":101,"threads_started":1}
I20260929 14:47:58.241717 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e938fa1e74c1467882333233e774f815" candidate_uuid: "528ede259700460c97efaef3ecfaa156" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" is_pre_election: true
I20260929 14:47:58.241896 8333 raft_consensus.cc:2454] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ede259700460c97efaef3ecfaa156 in term 0.
I20260929 14:47:58.241781 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e938fa1e74c1467882333233e774f815" candidate_uuid: "528ede259700460c97efaef3ecfaa156" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" is_pre_election: true
I20260929 14:47:58.242122 8052 raft_consensus.cc:2454] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 528ede259700460c97efaef3ecfaa156 in term 0.
I20260929 14:47:58.242309 8125 leader_election.cc:302] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 67e0c8f926a64ad6a6ab9daabccb36f6; no voters:
I20260929 14:47:58.242647 8414 raft_consensus.cc:2781] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:58.242728 8414 raft_consensus.cc:482] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:58.242785 8414 raft_consensus.cc:3037] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.243992 8414 raft_consensus.cc:504] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.244475 8414 leader_election.cc:288] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [CANDIDATE]: Term 1 election: Requested vote from peers 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:58.244798 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e938fa1e74c1467882333233e774f815" candidate_uuid: "528ede259700460c97efaef3ecfaa156" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca"
I20260929 14:47:58.244869 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e938fa1e74c1467882333233e774f815" candidate_uuid: "528ede259700460c97efaef3ecfaa156" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
I20260929 14:47:58.244951 8052 raft_consensus.cc:3037] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.245034 8333 raft_consensus.cc:3037] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.246065 8052 raft_consensus.cc:2454] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ede259700460c97efaef3ecfaa156 in term 1.
I20260929 14:47:58.246099 8333 raft_consensus.cc:2454] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 528ede259700460c97efaef3ecfaa156 in term 1.
I20260929 14:47:58.246364 8126 leader_election.cc:302] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528ede259700460c97efaef3ecfaa156, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:58.246537 8414 raft_consensus.cc:2781] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:58.246842 8414 raft_consensus.cc:686] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 1 LEADER]: Becoming Leader. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Running, Role: LEADER
I20260929 14:47:58.247155 8414 consensus_queue.cc:237] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.249975 7895 catalog_manager.cc:5495] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 reported cstate change: term changed from 0 to 1, leader changed from <none> to 528ede259700460c97efaef3ecfaa156 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "528ede259700460c97efaef3ecfaa156" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:58.288347 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.284051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.98.2:38213)
0929 14:47:58.284458 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.284468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.284482 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:47:58.284578 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.284580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.284650 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.284802 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:58.285066 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.285662 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.286549 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.286813 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.286928 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.287218 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:58.287835 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:58.287971 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.288033 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.288100 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":163,"threads_started":1}
I20260929 14:47:58.291772 8333 raft_consensus.cc:1261] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Refusing update from remote peer 528ede259700460c97efaef3ecfaa156: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.291962 8052 raft_consensus.cc:1261] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Refusing update from remote peer 528ede259700460c97efaef3ecfaa156: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.292500 8477 consensus_queue.cc:1035] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.292779 8414 consensus_queue.cc:1035] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.297469 8423 mvcc.cc:205] Tried to move back new op lower bound from 7334679667876077568 to 7334679667703091200. Current Snapshot: MvccSnapshot[applied={T|T < 7334679667876077568}]
I20260929 14:47:58.306267 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.306412 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.308553 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Sending DeleteTablet for 3 replicas of tablet e938fa1e74c1467882333233e774f815
I20260929 14:47:58.309095 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet e938fa1e74c1467882333233e774f815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52284
I20260929 14:47:58.309300 8430 tablet_replica.cc:330] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: stopping tablet replica
I20260929 14:47:58.309406 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet e938fa1e74c1467882333233e774f815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49430
I20260929 14:47:58.309433 8430 raft_consensus.cc:2229] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.309592 8430 raft_consensus.cc:2258] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.309593 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet e938fa1e74c1467882333233e774f815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44570
I20260929 14:47:58.309780 8432 tablet_replica.cc:330] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: stopping tablet replica
I20260929 14:47:58.309906 8432 raft_consensus.cc:2229] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:58.310204 8432 raft_consensus.cc:2258] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.310391 8431 tablet_replica.cc:330] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: stopping tablet replica
I20260929 14:47:58.310531 8431 raft_consensus.cc:2229] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.310688 8431 raft_consensus.cc:2258] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.310833 8430 ts_tablet_manager.cc:1905] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.311527 8431 ts_tablet_manager.cc:1905] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.312943 8432 ts_tablet_manager.cc:1905] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.313519 8430 ts_tablet_manager.cc:1918] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:58.313599 8430 log.cc:1196] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/e938fa1e74c1467882333233e774f815
I20260929 14:47:58.313866 8430 ts_tablet_manager.cc:1939] T e938fa1e74c1467882333233e774f815 P 9e0c33b7fa6f48ca8047965a6ab719ca: Deleting consensus metadata
I20260929 14:47:58.314896 7882 catalog_manager.cc:4841] TS 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527): tablet e938fa1e74c1467882333233e774f815 (table test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]) successfully deleted
I20260929 14:47:58.315307 8432 ts_tablet_manager.cc:1918] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:58.315390 8432 log.cc:1196] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/e938fa1e74c1467882333233e774f815
I20260929 14:47:58.315564 8432 ts_tablet_manager.cc:1939] T e938fa1e74c1467882333233e774f815 P 528ede259700460c97efaef3ecfaa156: Deleting consensus metadata
I20260929 14:47:58.316370 7883 catalog_manager.cc:4841] TS 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213): tablet e938fa1e74c1467882333233e774f815 (table test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]) successfully deleted
I20260929 14:47:58.317203 8431 ts_tablet_manager.cc:1918] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:47:58.317277 8431 log.cc:1196] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/e938fa1e74c1467882333233e774f815
I20260929 14:47:58.317453 8431 ts_tablet_manager.cc:1939] T e938fa1e74c1467882333233e774f815 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting consensus metadata
I20260929 14:47:58.317730 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:47:58.318171 7881 catalog_manager.cc:4841] TS 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035): tablet e938fa1e74c1467882333233e774f815 (table test-table [id=d46d7ea740bc44cdb13e7c97f7c17cb8]) successfully deleted
W20260929 14:47:58.318348 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:47:58.323222 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 (DEFAULT_TABLE table=test-table [id=c7cdef8ed8de4bda9c06d2a1df08e4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.323323 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 (DEFAULT_TABLE table=test-table [id=c7cdef8ed8de4bda9c06d2a1df08e4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.323522 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76562a862f4e46d6ba61afabd8eeeba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.323545 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76562a862f4e46d6ba61afabd8eeeba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.325897 8408 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: Bootstrap starting.
I20260929 14:47:58.326617 8408 tablet_bootstrap.cc:654] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.327092 8407 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Bootstrap starting.
I20260929 14:47:58.327886 8407 tablet_bootstrap.cc:654] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.328666 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 (DEFAULT_TABLE table=test-table [id=c7cdef8ed8de4bda9c06d2a1df08e4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:47:58.328949 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76562a862f4e46d6ba61afabd8eeeba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:47:58.329376 8407 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: No bootstrap required, opened a new log
I20260929 14:47:58.329463 8407 ts_tablet_manager.cc:1397] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:47:58.329895 8407 raft_consensus.cc:348] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.330021 8407 raft_consensus.cc:374] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.330054 8407 raft_consensus.cc:729] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e0c8f926a64ad6a6ab9daabccb36f6, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.330209 8407 consensus_queue.cc:260] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.330513 8407 ts_tablet_manager.cc:1428] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:47:58.331254 8409 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Bootstrap starting.
I20260929 14:47:58.332261 8409 tablet_bootstrap.cc:654] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Neither blocks nor log segments found. Creating new log.
I20260929 14:47:58.332612 8408 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: No bootstrap required, opened a new log
I20260929 14:47:58.332711 8408 ts_tablet_manager.cc:1397] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 14:47:58.333181 8408 raft_consensus.cc:348] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.333346 8408 raft_consensus.cc:374] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.333382 8408 raft_consensus.cc:729] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.333523 8408 consensus_queue.cc:260] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.333635 8409 tablet_bootstrap.cc:492] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: No bootstrap required, opened a new log
I20260929 14:47:58.333726 8409 ts_tablet_manager.cc:1397] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:47:58.333810 8408 ts_tablet_manager.cc:1428] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:58.334233 8409 raft_consensus.cc:348] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.334344 8409 raft_consensus.cc:374] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:47:58.334380 8409 raft_consensus.cc:729] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528ede259700460c97efaef3ecfaa156, State: Initialized, Role: FOLLOWER
I20260929 14:47:58.334504 8409 consensus_queue.cc:260] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.334769 8409 ts_tablet_manager.cc:1428] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:47:58.498209 8451 raft_consensus.cc:482] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:47:58.498433 8451 raft_consensus.cc:504] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.499069 8451 leader_election.cc:288] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:58.499635 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76562a862f4e46d6ba61afabd8eeeba7" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" is_pre_election: true
I20260929 14:47:58.499655 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76562a862f4e46d6ba61afabd8eeeba7" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156" is_pre_election: true
I20260929 14:47:58.499796 8333 raft_consensus.cc:2454] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:58.499922 8193 raft_consensus.cc:2454] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 0.
I20260929 14:47:58.500252 7985 leader_election.cc:302] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e0c8f926a64ad6a6ab9daabccb36f6, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:58.500494 8451 raft_consensus.cc:2781] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:47:58.500573 8451 raft_consensus.cc:482] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:47:58.500604 8451 raft_consensus.cc:3037] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.501892 8451 raft_consensus.cc:504] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.502305 8451 leader_election.cc:288] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Requested vote from peers 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213), 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035)
I20260929 14:47:58.502743 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76562a862f4e46d6ba61afabd8eeeba7" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528ede259700460c97efaef3ecfaa156"
I20260929 14:47:58.502776 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76562a862f4e46d6ba61afabd8eeeba7" candidate_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6"
I20260929 14:47:58.502923 8193 raft_consensus.cc:3037] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.502944 8333 raft_consensus.cc:3037] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:47:58.504024 8333 raft_consensus.cc:2454] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:58.504026 8193 raft_consensus.cc:2454] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e0c33b7fa6f48ca8047965a6ab719ca in term 1.
I20260929 14:47:58.504462 7985 leader_election.cc:302] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e0c8f926a64ad6a6ab9daabccb36f6, 9e0c33b7fa6f48ca8047965a6ab719ca; no voters:
I20260929 14:47:58.504778 8451 raft_consensus.cc:2781] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:47:58.504932 8451 raft_consensus.cc:686] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Becoming Leader. State: Replica: 9e0c33b7fa6f48ca8047965a6ab719ca, State: Running, Role: LEADER
I20260929 14:47:58.505110 8451 consensus_queue.cc:237] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } }
I20260929 14:47:58.507027 7896 catalog_manager.cc:5495] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e0c33b7fa6f48ca8047965a6ab719ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 } health_report { overall_health: UNKNOWN } } }
I20260929 14:47:58.553836 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.548609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.1:37527)
0929 14:47:58.548780 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:47:58.548785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:47:58.548804 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:47:58.548900 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:47:58.548903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:47:58.548998 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:47:58.549119 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:47:58.549722 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.550609 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.552079 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:47:58.552570 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:47:58.552715 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:47:58.553117 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:47:58.553252 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:47:58.553341 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:47:58.553519 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:47:58.553606 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:47:58.562633 8333 raft_consensus.cc:1261] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.562821 8193 raft_consensus.cc:1261] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Refusing update from remote peer 9e0c33b7fa6f48ca8047965a6ab719ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:47:58.563228 8451 consensus_queue.cc:1035] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e0c8f926a64ad6a6ab9daabccb36f6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.563616 8489 consensus_queue.cc:1035] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "528ede259700460c97efaef3ecfaa156" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:47:58.618650 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.618896 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:47:58.621953 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 043409dd056040d29e35b87091f36a37: Sending DeleteTablet for 3 replicas of tablet 76562a862f4e46d6ba61afabd8eeeba7
I20260929 14:47:58.622723 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52284
I20260929 14:47:58.622993 8430 tablet_replica.cc:330] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca: stopping tablet replica
I20260929 14:47:58.623128 8430 raft_consensus.cc:2229] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:47:58.623179 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44570
I20260929 14:47:58.623377 8430 raft_consensus.cc:2258] T 76562a862f4e46d6ba61afabd8eeeba7 P 9e0c33b7fa6f48ca8047965a6ab719ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.623440 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 76562a862f4e46d6ba61afabd8eeeba7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:47:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49430
I20260929 14:47:58.623685 8431 tablet_replica.cc:330] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: stopping tablet replica
I20260929 14:47:58.623819 8431 raft_consensus.cc:2229] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.623977 8431 raft_consensus.cc:2258] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.624140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7968
I20260929 14:47:58.625368 8432 tablet_replica.cc:330] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: stopping tablet replica
I20260929 14:47:58.625547 8432 raft_consensus.cc:2229] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:47:58.625699 8432 raft_consensus.cc:2258] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:47:58.627683 8432 ts_tablet_manager.cc:1905] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.627683 8431 ts_tablet_manager.cc:1905] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:47:58.630656 8432 ts_tablet_manager.cc:1918] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:58.630656 8431 ts_tablet_manager.cc:1918] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:47:58.630759 8431 log.cc:1196] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/76562a862f4e46d6ba61afabd8eeeba7
I20260929 14:47:58.630759 8432 log.cc:1196] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/76562a862f4e46d6ba61afabd8eeeba7
I20260929 14:47:58.631042 8432 ts_tablet_manager.cc:1939] T 76562a862f4e46d6ba61afabd8eeeba7 P 528ede259700460c97efaef3ecfaa156: Deleting consensus metadata
I20260929 14:47:58.631042 8431 ts_tablet_manager.cc:1939] T 76562a862f4e46d6ba61afabd8eeeba7 P 67e0c8f926a64ad6a6ab9daabccb36f6: Deleting consensus metadata
I20260929 14:47:58.632005 7881 catalog_manager.cc:4841] TS 67e0c8f926a64ad6a6ab9daabccb36f6 (127.0.98.3:40035): tablet 76562a862f4e46d6ba61afabd8eeeba7 (table test-table [id=c7cdef8ed8de4bda9c06d2a1df08e4e3]) successfully deleted
I20260929 14:47:58.632336 7883 catalog_manager.cc:4841] TS 528ede259700460c97efaef3ecfaa156 (127.0.98.2:38213): tablet 76562a862f4e46d6ba61afabd8eeeba7 (table test-table [id=c7cdef8ed8de4bda9c06d2a1df08e4e3]) successfully deleted
W20260929 14:47:58.634889 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37527 (error 108)
I20260929 14:47:58.635063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8108
I20260929 14:47:58.637863 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:47:58.635669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37527 (local address 127.0.0.1:52336)
0929 14:47:58.637585 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0929 14:47:58.637696 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1790,"thread_start_us":61,"threads_started":1}
W20260929 14:47:58.638074 7882 catalog_manager.cc:4584] TS 9e0c33b7fa6f48ca8047965a6ab719ca (127.0.98.1:37527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76562a862f4e46d6ba61afabd8eeeba7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37527: connect: Connection refused (error 111)
I20260929 14:47:58.643747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8248
I20260929 14:47:58.651013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 7864
2026-09-29T14:47:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3400 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[8498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[8498](info): set up 2 sockets
Sep 29 14:47:58 dist-test-slave-5hd1 krb5kdc[8498](info): commencing operation
krb5kdc: starting...
W20260929 14:47:59.709142 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:47:59 dist-test-slave-5hd1 krb5kdc[8498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:47:59.843775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:47:59.973770 8511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:47:59.974180 8511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:47:59.974265 8511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:47:59.979238 8511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:47:59.979384 8511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:47:59.979425 8511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:47:59.979458 8511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:47:59.979518 8511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:47:59.985900 8511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:47:59.987406 8511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:47:59.988755 8511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:47:59.999169 8517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:59.999214 8516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:47:59.999224 8519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:47:59.999817 8511 server_base.cc:1034] running on GCE node
I20260929 14:48:00.000612 8511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:00.002030 8511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:00.003247 8511 hybrid_clock.cc:648] HybridClock initialized: now 1790693280003208 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T14:48:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1358 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:48:00 dist-test-slave-5hd1 krb5kdc[8524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:48:00 dist-test-slave-5hd1 krb5kdc[8524](info): set up 2 sockets
Sep 29 14:48:00 dist-test-slave-5hd1 krb5kdc[8524](info): commencing operation
krb5kdc: starting...
W20260929 14:48:01.065354 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:48:01.204223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:01.343863 8537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:01.344367 8537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:01.344424 8537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:01.349232 8537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:48:01.349354 8537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:01.349377 8537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:01.349396 8537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:48:01.349414 8537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:48:01.356498 8537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:01.358129 8537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:01.359536 8537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:01.371599 8542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.371620 8543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.371611 8545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:01.371963 8537 server_base.cc:1034] running on GCE node
I20260929 14:48:01.372845 8537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:01.374148 8537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:01.375373 8537 hybrid_clock.cc:648] HybridClock initialized: now 1790693281375357 us; error 44 us; skew 500 ppm
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:01.378634 8537 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:48:01.379673 8537 webserver.cc:460] Webserver started at http://127.0.98.62:35989/ using document root <none> and password file <none>
I20260929 14:48:01.380396 8537 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:01.380491 8537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:01.380748 8537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:01.382969 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "1c7137df6fa040aaa7e5d4147a79a7b3"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "d0cb8b6a0bfd5085d1631b4bdeb3cb13"
server_key_iv: "21bb1b9866e30ab2663daa843b46a0af"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.383610 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c7137df6fa040aaa7e5d4147a79a7b3"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "d0cb8b6a0bfd5085d1631b4bdeb3cb13"
server_key_iv: "21bb1b9866e30ab2663daa843b46a0af"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.388398 8537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 14:48:01.391355 8552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:01.393226 8537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:01.393409 8537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "1c7137df6fa040aaa7e5d4147a79a7b3"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "d0cb8b6a0bfd5085d1631b4bdeb3cb13"
server_key_iv: "21bb1b9866e30ab2663daa843b46a0af"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.393553 8537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:01.418469 8537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:01.422111 8537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:01.422423 8537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:01.431145 8537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39383
I20260929 14:48:01.431231 8604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39383 every 8 connection(s)
I20260929 14:48:01.432380 8537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:48:01.433018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8537
I20260929 14:48:01.433139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:48:01.433480 392 external_mini_cluster.cc:1428] Setting key fae1a14021d77aaffb493161f499e139
I20260929 14:48:01.436663 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:01.445425 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Bootstrap starting.
I20260929 14:48:01.449184 8605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:01.450350 8605 log.cc:824] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:01.452714 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:01.435158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.98.62:39383)
0929 14:48:01.435771 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:01.435789 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:01.436275 (+ 486us) server_negotiation.cc:407] Connection header received
0929 14:48:01.437180 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:01.437203 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:01.437628 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:01.438020 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:01.439137 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.440376 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.441716 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.442329 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.442500 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:01.446059 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:01.446067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:01.446090 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:01.446138 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:01.448577 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:01.449213 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:01.449221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:01.449228 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:01.449408 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:01.449814 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:01.449818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:01.449819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:01.450500 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:01.450722 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:01.451058 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 14:48:01.451498 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":210,"threads_started":1}
I20260929 14:48:01.453615 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: No bootstrap required, opened a new log
I20260929 14:48:01.458168 8605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:01.458482 8605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:01.458549 8605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7137df6fa040aaa7e5d4147a79a7b3, State: Initialized, Role: FOLLOWER
I20260929 14:48:01.459179 8605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:01.459360 8605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:48:01.459434 8605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:48:01.459553 8605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:01.460867 8605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:01.461342 8605 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7137df6fa040aaa7e5d4147a79a7b3; no voters:
I20260929 14:48:01.461808 8605 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:48:01.461915 8610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:48:01.462126 8610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7137df6fa040aaa7e5d4147a79a7b3, State: Running, Role: LEADER
I20260929 14:48:01.462498 8610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:01.463469 8605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:48:01.464816 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260929 14:48:01.465070 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:01.465473 8611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7137df6fa040aaa7e5d4147a79a7b3. Latest consensus state: current_term: 1 leader_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260929 14:48:01.465616 8611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:01.465780 8619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:48:01.469496 8619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:48:01.476824 8619 catalog_manager.cc:1347] Generated new cluster ID: bff0090873be451b9ba1eb8902a8b96a
I20260929 14:48:01.476964 8619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:48:01.492372 8619 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:48:01.493839 8619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:48:01.500985 8619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Generated new TSK 0
I20260929 14:48:01.501830 8619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:48:01.545148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:01.683620 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:01.683996 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:01.684054 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:01.688715 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:01.688817 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:01.688951 8631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:48:01.695252 8631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:01.696851 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:01.698421 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:01.710450 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.710400 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.710450 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:01.710788 8631 server_base.cc:1034] running on GCE node
I20260929 14:48:01.711380 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:01.712275 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:01.713501 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790693281713478 us; error 52 us; skew 500 ppm
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:01.722791 8631 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:48:01.724228 8631 webserver.cc:460] Webserver started at http://127.0.98.1:37849/ using document root <none> and password file <none>
I20260929 14:48:01.725013 8631 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:01.725122 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:01.725417 8631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:01.728112 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b2ad5344b394af7b3633504a2a74708"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "3019de04803634f4a33bdf02c0e9a867"
server_key_iv: "a0a68fa9ce379d3d9999c3d82f941919"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.728904 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b2ad5344b394af7b3633504a2a74708"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "3019de04803634f4a33bdf02c0e9a867"
server_key_iv: "a0a68fa9ce379d3d9999c3d82f941919"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.733930 8631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:48:01.736994 8646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:01.738644 8631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:48:01.738781 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "0b2ad5344b394af7b3633504a2a74708"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "3019de04803634f4a33bdf02c0e9a867"
server_key_iv: "a0a68fa9ce379d3d9999c3d82f941919"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.738955 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:01.761667 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:01.765621 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:01.765833 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:01.766495 8631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:01.767356 8631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:01.767428 8631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:01.767500 8631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:01.767539 8631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:01.777702 8631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45807
I20260929 14:48:01.777782 8759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45807 every 8 connection(s)
I20260929 14:48:01.778684 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:48:01.784449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8631
I20260929 14:48:01.784611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:48:01.784951 392 external_mini_cluster.cc:1428] Setting key 1a33f42eaa1c1ede8911f528eac3824d
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:01.795009 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:01.781138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52591 (local address 127.0.98.62:39383)
0929 14:48:01.781334 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:01.781338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:01.782351 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 14:48:01.783887 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:01.783890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:01.783950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:01.784055 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:01.785448 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.786018 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.787170 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.787406 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.787515 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:01.790451 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:01.790456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:01.790459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:01.790490 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:01.792368 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:01.793109 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:01.793112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:01.793114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:01.793206 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:01.793635 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:01.793638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:01.793639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:01.793842 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:01.793925 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:01.794709 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 14:48:01.794858 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 14:48:01.795989 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:01.781604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:52591)
0929 14:48:01.782176 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:01.782225 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:01.783645 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:01.784291 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:01.784299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:01.784783 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:01.785305 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:01.785318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.786154 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.786159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:01.787040 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:01.787049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:01.787558 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:01.787562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:01.787736 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:01.788508 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:01.788525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:01.790221 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:01.792502 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:01.792511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:01.792529 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:01.792905 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:01.793323 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:01.793330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:01.793334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:01.793525 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:01.793963 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:01.793972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:01.794292 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 14:48:01.794529 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 14:48:01.794876 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":163,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:48:01.797495 8760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260929 14:48:01.797884 8760 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:01.798704 8760 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:48:01.800894 8569 ts_manager.cc:194] Registered new tserver with Master: 0b2ad5344b394af7b3633504a2a74708 (127.0.98.1:45807)
I20260929 14:48:01.802261 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:48:01.820168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:01.957038 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:01.957397 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:01.957484 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:01.961947 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:01.962055 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:01.962155 8765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:48:01.968549 8765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:01.970000 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:01.971278 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:01.982847 8771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.982847 8773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:01.982858 8770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:01.983561 8765 server_base.cc:1034] running on GCE node
I20260929 14:48:01.984067 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:01.984910 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:01.986119 8765 hybrid_clock.cc:648] HybridClock initialized: now 1790693281986078 us; error 61 us; skew 500 ppm
Sep 29 14:48:01 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:01.989524 8765 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:48:01.990635 8765 webserver.cc:460] Webserver started at http://127.0.98.2:34561/ using document root <none> and password file <none>
I20260929 14:48:01.991333 8765 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:01.991426 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:01.991704 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:01.994309 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd6df7573e214be991ee1612ca25e344"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "8791172a9de7daea8cdab4ce7beb2bae"
server_key_iv: "65addbdea29471ed06c5139e002cd102"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.994990 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd6df7573e214be991ee1612ca25e344"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "8791172a9de7daea8cdab4ce7beb2bae"
server_key_iv: "65addbdea29471ed06c5139e002cd102"
server_key_version: "encryptionkey@0"
I20260929 14:48:01.999815 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 14:48:02.003105 8780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.004814 8765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 14:48:02.004976 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "dd6df7573e214be991ee1612ca25e344"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "8791172a9de7daea8cdab4ce7beb2bae"
server_key_iv: "65addbdea29471ed06c5139e002cd102"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.005110 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:02.019070 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:02.023192 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:02.023448 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:02.024289 8765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:02.025365 8765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:02.025440 8765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.025518 8765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:02.025568 8765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.037717 8765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46769
I20260929 14:48:02.037761 8893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46769 every 8 connection(s)
I20260929 14:48:02.039101 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:48:02.048902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8765
I20260929 14:48:02.049083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:48:02.049446 392 external_mini_cluster.cc:1428] Setting key adbb3d00b7cdf0c0a6f09ee451c10184
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:02.055757 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.041555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54591 (local address 127.0.98.62:39383)
0929 14:48:02.041774 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:02.041780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:02.043085 (+ 1305us) server_negotiation.cc:407] Connection header received
0929 14:48:02.044732 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:02.044735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:02.044800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:02.044903 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:02.046157 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.046754 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.047874 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.048116 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.048283 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.051407 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:02.051410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:02.051413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:02.051435 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:02.053026 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:02.053832 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:02.053838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:02.053839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:02.053935 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:02.054337 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:02.054342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:02.054343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:02.054612 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:02.054732 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:02.055369 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 14:48:02.055527 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 14:48:02.056879 8896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.042109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:54591)
0929 14:48:02.042855 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:02.042939 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:02.044496 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:02.045080 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:02.045088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:02.045466 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:02.045995 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.046009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.046871 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.046873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.047750 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.047758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.048315 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.048318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.048451 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.049210 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:02.049232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:02.051195 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:02.053194 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:02.053206 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:02.053230 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:02.053713 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:02.054049 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:02.054052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:02.054053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:02.054228 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:02.054737 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:02.054743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:02.055056 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 14:48:02.055307 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 14:48:02.055705 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":138,"threads_started":1}
I20260929 14:48:02.058477 8894 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260929 14:48:02.058848 8894 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:02.059583 8894 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:48:02.061316 8569 ts_manager.cc:194] Registered new tserver with Master: dd6df7573e214be991ee1612ca25e344 (127.0.98.2:46769)
I20260929 14:48:02.062206 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:48:02.079658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:02.206117 8899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:02.206408 8899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:02.206453 8899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:02.210865 8899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:02.210947 8899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:02.211009 8899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:48:02.216809 8899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:02.218084 8899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:02.219199 8899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:02.230669 8904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:02.231042 8905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:02.231025 8899 server_base.cc:1034] running on GCE node
W20260929 14:48:02.230744 8907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:02.231782 8899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:02.232605 8899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:02.233788 8899 hybrid_clock.cc:648] HybridClock initialized: now 1790693282233742 us; error 65 us; skew 500 ppm
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:02.236829 8899 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:48:02.237746 8899 webserver.cc:460] Webserver started at http://127.0.98.3:36961/ using document root <none> and password file <none>
I20260929 14:48:02.238301 8899 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:02.238363 8899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:02.238534 8899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:02.240635 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "0524504696ae4a66940ecb463e8057a4"
format_stamp: "Formatted at 2026-09-29 14:48:02 on dist-test-slave-5hd1"
server_key: "76814a677160b80babb27a60b143fbb4"
server_key_iv: "8014465963c3901a727236c1c98377c6"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.241163 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0524504696ae4a66940ecb463e8057a4"
format_stamp: "Formatted at 2026-09-29 14:48:02 on dist-test-slave-5hd1"
server_key: "76814a677160b80babb27a60b143fbb4"
server_key_iv: "8014465963c3901a727236c1c98377c6"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.245908 8899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:48:02.249042 8914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.250582 8899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:48:02.250715 8899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "0524504696ae4a66940ecb463e8057a4"
format_stamp: "Formatted at 2026-09-29 14:48:02 on dist-test-slave-5hd1"
server_key: "76814a677160b80babb27a60b143fbb4"
server_key_iv: "8014465963c3901a727236c1c98377c6"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.250810 8899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:02.269243 8899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:02.272758 8899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:02.272926 8899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:02.273489 8899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:02.274408 8899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:02.274461 8899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.274501 8899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:02.274573 8899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.285121 8899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41275
I20260929 14:48:02.285149 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41275 every 8 connection(s)
I20260929 14:48:02.286060 8899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:48:02.287384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 8899
I20260929 14:48:02.287503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:48:02.287835 392 external_mini_cluster.cc:1428] Setting key 5cab604d5b4a92218198504a9b69d19e
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:02.303040 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.288639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40825 (local address 127.0.98.62:39383)
0929 14:48:02.288785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:02.288790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:02.290289 (+ 1499us) server_negotiation.cc:407] Connection header received
0929 14:48:02.291788 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:02.291791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:02.291843 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:02.291923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:02.293118 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.293653 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.294780 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.295000 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.295107 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.298184 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:02.298189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:02.298191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:02.298228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:02.300160 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:02.300891 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:02.300895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:02.300897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:02.300999 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:02.301600 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:02.301604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:02.301606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:02.301832 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:02.302031 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:02.302664 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 14:48:02.302811 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:48:02.304144 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.289390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:40825)
0929 14:48:02.290049 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:02.290145 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:02.291564 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:02.292090 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:02.292097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:02.292504 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:02.292982 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.292994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.293779 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.293785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.294659 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.294668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.295143 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.295147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.295369 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.296324 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:02.296343 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:02.297987 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:02.300460 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:02.300465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:02.300477 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:02.300758 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:02.301156 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:02.301160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:02.301163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:02.301473 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:02.302031 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:02.302041 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:02.302368 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 14:48:02.302621 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 14:48:02.303066 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":219,"threads_started":1}
I20260929 14:48:02.305506 9028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260929 14:48:02.305804 9028 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:02.306365 9028 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260929 14:48:02.307574 8569 ts_manager.cc:194] Registered new tserver with Master: 0524504696ae4a66940ecb463e8057a4 (127.0.98.3:41275)
I20260929 14:48:02.308477 8569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40825
I20260929 14:48:02.314458 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:48:02.403878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8631
I20260929 14:48:02.410563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8765
I20260929 14:48:02.416829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8899
I20260929 14:48:02.423599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 8537
I20260929 14:48:02.431712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--webserver_interface=127.0.98.62
--webserver_port=35989
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:02.549178 9037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:02.549489 9037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:02.549532 9037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:02.554025 9037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:48:02.554117 9037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:02.554384 9037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:02.554450 9037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:48:02.554570 9037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:48:02.562793 9037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:02.563947 9037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:02.565022 9037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:02.573993 9043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:02.574043 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:02.573993 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:02.574288 9037 server_base.cc:1034] running on GCE node
I20260929 14:48:02.574918 9037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:02.576040 9037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:02.577253 9037 hybrid_clock.cc:648] HybridClock initialized: now 1790693282577231 us; error 50 us; skew 500 ppm
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:02.580082 9037 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:48:02.581125 9037 webserver.cc:460] Webserver started at http://127.0.98.62:35989/ using document root <none> and password file <none>
I20260929 14:48:02.581645 9037 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:02.581717 9037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:02.585705 9037 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:02.588074 9052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.589533 9037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:48:02.589685 9037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "1c7137df6fa040aaa7e5d4147a79a7b3"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "d0cb8b6a0bfd5085d1631b4bdeb3cb13"
server_key_iv: "21bb1b9866e30ab2663daa843b46a0af"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.590234 9037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:02.639585 9037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:02.642831 9037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:02.643045 9037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:02.651038 9037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39383
I20260929 14:48:02.651098 9104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39383 every 8 connection(s)
I20260929 14:48:02.652105 9037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:48:02.657673 9105 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:48:02.659829 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Bootstrap starting.
I20260929 14:48:02.661033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9037
I20260929 14:48:02.661525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45807
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39383
--webserver_port=37849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:48:02.680605 9105 log.cc:824] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:02.688854 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:48:02.689329 9105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Bootstrap complete.
I20260929 14:48:02.694754 9105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:02.695302 9105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7137df6fa040aaa7e5d4147a79a7b3, State: Initialized, Role: FOLLOWER
I20260929 14:48:02.695995 9105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:02.696164 9105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:48:02.696273 9105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:48:02.696401 9105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:48:02.698858 9105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:02.699326 9105 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7137df6fa040aaa7e5d4147a79a7b3; no voters:
I20260929 14:48:02.699759 9105 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:48:02.700007 9109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:48:02.700403 9109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [term 2 LEADER]: Becoming Leader. State: Replica: 1c7137df6fa040aaa7e5d4147a79a7b3, State: Running, Role: LEADER
I20260929 14:48:02.700829 9109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260929 14:48:02.701363 9105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:48:02.702394 9110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260929 14:48:02.702541 9110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:02.702895 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7137df6fa040aaa7e5d4147a79a7b3. Latest consensus state: current_term: 2 leader_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7137df6fa040aaa7e5d4147a79a7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260929 14:48:02.702991 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:02.703861 9118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:48:02.707811 9118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:48:02.708492 9118 catalog_manager.cc:1259] Loaded cluster ID: bff0090873be451b9ba1eb8902a8b96a
I20260929 14:48:02.708557 9118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:48:02.710894 9118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:48:02.711542 9118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c7137df6fa040aaa7e5d4147a79a7b3: Loaded TSK: 0
I20260929 14:48:02.712787 9118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:48:02.778328 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:02.778612 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:02.778658 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:02.782981 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:02.783051 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:02.783111 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:48:02.788620 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37849
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:02.789672 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:02.790696 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:02.800985 9131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:02.801302 9107 server_base.cc:1034] running on GCE node
W20260929 14:48:02.800968 9134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:02.800968 9132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:02.802004 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:02.802763 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:02.803953 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790693282803925 us; error 46 us; skew 500 ppm
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:02.807219 9107 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:48:02.808218 9107 webserver.cc:460] Webserver started at http://127.0.98.1:37849/ using document root <none> and password file <none>
I20260929 14:48:02.808795 9107 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:02.808908 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:02.813156 9107 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:02.815698 9141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.817253 9107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:48:02.817418 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "0b2ad5344b394af7b3633504a2a74708"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "3019de04803634f4a33bdf02c0e9a867"
server_key_iv: "a0a68fa9ce379d3d9999c3d82f941919"
server_key_version: "encryptionkey@0"
I20260929 14:48:02.818009 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:02.842290 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:02.846215 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:02.846457 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:02.847198 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:02.848100 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:02.848212 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.848295 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:02.848349 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:02.860744 9107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45807
I20260929 14:48:02.860819 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45807 every 8 connection(s)
I20260929 14:48:02.861908 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:48:02.869402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9107
I20260929 14:48:02.870064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46769
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39383
--webserver_port=34561
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:02.878084 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:50325)
0929 14:48:02.865307 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:02.865360 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:02.866655 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:02.867958 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:02.867964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:02.868480 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:02.868991 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.869004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.871170 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.871176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.872515 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.872526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.873421 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.873425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.873600 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.874347 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:02.874364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:02.877043 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":153,"threads_started":1}
W20260929 14:48:02.878073 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:02.864651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50325 (local address 127.0.98.62:39383)
0929 14:48:02.865183 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:02.865209 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:02.865498 (+ 289us) server_negotiation.cc:407] Connection header received
0929 14:48:02.867088 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:02.867103 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:02.867476 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:02.867817 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:02.869207 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.870959 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.872702 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.873233 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.873449 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.877284 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50325: BlockingRecv error: recv got EOF from 127.0.98.1:50325 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":185,"threads_started":1}
W20260929 14:48:02.878744 9255 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39383 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39383: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:02.884688 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.879416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:46803)
0929 14:48:02.879655 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:02.879668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:02.879782 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:02.880037 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:02.880040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:02.880259 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:02.880350 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.880356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.881105 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.881112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.881662 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.881670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.882131 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.882134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.882225 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.882805 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:02.882815 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:02.884523 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260929 14:48:02.884780 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:02.879426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46803 (local address 127.0.98.62:39383)
0929 14:48:02.879561 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:02.879564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:02.879734 (+ 170us) server_negotiation.cc:407] Connection header received
0929 14:48:02.879883 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:02.879886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:02.879939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:02.880022 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:02.880461 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.880999 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.881782 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.882017 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.882126 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.884648 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46803: BlockingRecv error: recv got EOF from 127.0.98.1:46803 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:02 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:02.890314 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:02.885599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:38465)
0929 14:48:02.885752 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:02.885768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:02.885847 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:02.886092 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:02.886095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:02.886241 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:02.886305 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.886310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.886997 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.887004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.887508 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:02.887515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.887946 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.887950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:02.888026 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.888578 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:02.888589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:02.890181 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 14:48:02.890403 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:02.885686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38465 (local address 127.0.98.62:39383)
0929 14:48:02.885796 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:02.885802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:02.885817 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:48:02.885947 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:02.885950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:02.885996 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:02.886079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:02.886414 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.886892 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.887623 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:02.887834 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:02.887942 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:02.890290 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38465: BlockingRecv error: recv got EOF from 127.0.98.1:38465 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 14:48:02.890784 9255 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:48:03.004451 9259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:03.004841 9259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:03.004930 9259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:03.010191 9259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:03.010325 9259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:03.010439 9259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:48:03.016752 9259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34561
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:03.018282 9259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:03.019599 9259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:03.031697 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:03.031766 9267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:03.031694 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:03.032017 9259 server_base.cc:1034] running on GCE node
I20260929 14:48:03.032591 9259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:03.033448 9259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:03.034683 9259 hybrid_clock.cc:648] HybridClock initialized: now 1790693283034662 us; error 41 us; skew 500 ppm
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:03.038172 9259 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:48:03.039332 9259 webserver.cc:460] Webserver started at http://127.0.98.2:34561/ using document root <none> and password file <none>
I20260929 14:48:03.040020 9259 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:03.040108 9259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:03.045171 9259 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:48:03.047981 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:03.049713 9259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:03.049908 9259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "dd6df7573e214be991ee1612ca25e344"
format_stamp: "Formatted at 2026-09-29 14:48:01 on dist-test-slave-5hd1"
server_key: "8791172a9de7daea8cdab4ce7beb2bae"
server_key_iv: "65addbdea29471ed06c5139e002cd102"
server_key_version: "encryptionkey@0"
I20260929 14:48:03.050595 9259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:03.073714 9259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:03.077710 9259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:03.077967 9259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:03.078758 9259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:03.079982 9259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:03.080078 9259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:03.080214 9259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:03.080274 9259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 14:48:03.092254 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46769 every 8 connection(s)
I20260929 14:48:03.092257 9259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46769
I20260929 14:48:03.093766 9259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:48:03.099098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9259
I20260929 14:48:03.099684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41275
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39383
--webserver_port=36961
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:48:03.109962 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.096256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54239 (local address 127.0.98.62:39383)
0929 14:48:03.096449 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.096455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.097639 (+ 1184us) server_negotiation.cc:407] Connection header received
0929 14:48:03.099043 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.099047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.099113 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.099242 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.101527 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.102519 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.104066 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.104533 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.104686 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.109750 (+ 5064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54239: BlockingRecv error: recv got EOF from 127.0.98.2:54239 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:48:03.110699 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.096857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:54239)
0929 14:48:03.097453 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.097503 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.098811 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.099574 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.099588 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.100249 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.101256 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.101283 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.102703 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.102706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.103923 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.103935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.104778 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.104784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.104970 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.105712 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.105731 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.109498 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":159,"threads_started":1}
W20260929 14:48:03.111536 9388 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39383 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39383: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:03.118989 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.112174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:49733)
0929 14:48:03.112358 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.112376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.112496 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.112854 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.112857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.113099 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.113226 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.113233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.114528 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.114531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.115429 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.115442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.116122 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.116125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.116277 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.116872 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.116884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.118863 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 14:48:03.119169 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.112327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49733 (local address 127.0.98.62:39383)
0929 14:48:03.112496 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.112501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.112522 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:48:03.112609 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.112612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.112672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.112798 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.113368 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.114384 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.115572 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.115976 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.116091 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.119031 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49733: BlockingRecv error: recv got EOF from 127.0.98.2:49733 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:03.125335 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.119794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:48367)
0929 14:48:03.119915 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.119931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.120034 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.120382 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.120385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.120556 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.120632 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.120637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.121386 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.121392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.121982 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.121991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.122440 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.122444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.122532 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.123073 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.123094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.125170 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 14:48:03.125566 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.119934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48367 (local address 127.0.98.62:39383)
0929 14:48:03.120126 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.120130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.120142 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:48:03.120223 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.120226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.120278 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.120365 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.120747 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.121279 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.122112 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.122330 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.122437 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.125305 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48367: BlockingRecv error: recv got EOF from 127.0.98.2:48367 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 14:48:03.125618 9388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:48:03.226611 9391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:03.226886 9391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:03.226969 9391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:03.231477 9391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:03.231566 9391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:03.231628 9391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:48:03.238971 9391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36961
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:03.240360 9391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:03.241444 9391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:03.251854 9396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:03.251853 9399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:03.252027 9397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:03.252385 9391 server_base.cc:1034] running on GCE node
I20260929 14:48:03.252796 9391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:03.253415 9391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:03.254650 9391 hybrid_clock.cc:648] HybridClock initialized: now 1790693283254627 us; error 45 us; skew 500 ppm
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:03.257961 9391 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:48:03.258982 9391 webserver.cc:460] Webserver started at http://127.0.98.3:36961/ using document root <none> and password file <none>
I20260929 14:48:03.259588 9391 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:03.259699 9391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:03.263971 9391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:48:03.266487 9406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:03.267970 9391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:48:03.268134 9391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "0524504696ae4a66940ecb463e8057a4"
format_stamp: "Formatted at 2026-09-29 14:48:02 on dist-test-slave-5hd1"
server_key: "76814a677160b80babb27a60b143fbb4"
server_key_iv: "8014465963c3901a727236c1c98377c6"
server_key_version: "encryptionkey@0"
I20260929 14:48:03.268832 9391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:03.286715 9391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:03.290418 9391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:03.290652 9391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:03.291301 9391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:03.292167 9391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:03.292313 9391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:03.292382 9391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:03.292429 9391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:03.303385 9391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41275
I20260929 14:48:03.303407 9519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41275 every 8 connection(s)
I20260929 14:48:03.304692 9391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:48:03.310860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9391
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:48:03.317719 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.306700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53079 (local address 127.0.98.62:39383)
0929 14:48:03.306886 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.306892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.307894 (+ 1002us) server_negotiation.cc:407] Connection header received
0929 14:48:03.309523 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.309526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.309581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.309690 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.311178 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.311930 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.313330 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.313613 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.313697 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.317548 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53079: BlockingRecv error: recv got EOF from 127.0.98.3:53079 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260929 14:48:03.318600 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.307138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:53079)
0929 14:48:03.307697 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.307746 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.309263 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.309892 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.309899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.310306 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.310996 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.311020 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.312141 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.312147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.313188 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.313199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.313793 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.313798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.314017 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.314755 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.314772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.317398 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":153,"threads_started":1}
W20260929 14:48:03.319203 9520 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39383 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39383: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 14:48:03.322186 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.312038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.0.98.62:39383)
0929 14:48:03.312362 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.312367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.312381 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:48:03.312503 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.312506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.312553 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.312645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.313438 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.314239 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.315292 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.315541 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.315657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.317235 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:03.317240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:03.317262 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:03.317289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:03.319487 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:03.320334 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:03.320339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:03.320344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:03.320450 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:03.320796 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:03.320799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:03.320801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:03.321239 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:03.321446 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:03.321735 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 14:48:03.321967 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":155,"threads_started":1}
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:03.325525 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.319750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:57151)
0929 14:48:03.319899 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.319917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.320006 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.320269 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.320273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.320410 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.320552 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.320559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.321515 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.321519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.322341 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.322351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.322896 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.322899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.322973 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.323467 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.323478 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.325423 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 14:48:03.325680 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.319828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57151 (local address 127.0.98.62:39383)
0929 14:48:03.319940 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.319943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.319960 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:48:03.320088 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.320090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.320122 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.320246 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.320656 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.321406 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.322447 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.322768 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.322886 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.325534 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57151: BlockingRecv error: recv got EOF from 127.0.98.3:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:03.331284 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.326138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:45413)
0929 14:48:03.326261 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.326273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.326369 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.326665 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.326668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.326815 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.326881 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.326886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.327640 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.327646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.328234 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.328242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.328706 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.328709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.328788 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.329285 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.329296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.331111 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 14:48:03.331362 9257 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.326247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45413 (local address 127.0.98.62:39383)
0929 14:48:03.326450 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.326455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.326467 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:48:03.326511 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.326513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.326550 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.326652 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.326994 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.327524 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.328344 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.328590 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.328704 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.331240 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45413: BlockingRecv error: recv got EOF from 127.0.98.3:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 14:48:03.331621 9520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:03 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:03.897838 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:03.891411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:60007)
0929 14:48:03.891701 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:03.891716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:03.891818 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:03.892276 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:03.892279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:03.892469 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:03.892623 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.892629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.893458 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.893460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.894137 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:03.894146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.894649 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.894651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:03.894744 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.895362 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:03.895374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:03.897613 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
W20260929 14:48:03.897889 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:03.891588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60007 (local address 127.0.98.62:39383)
0929 14:48:03.891908 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:03.891912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:03.891926 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:48:03.891977 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:03.891980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:03.892032 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:03.892130 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:03.892754 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.893349 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.894247 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:03.894510 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:03.894595 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:03.897754 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60007: BlockingRecv error: recv got EOF from 127.0.98.1:60007 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":144,"threads_started":1}
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:04.133338 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:04.126411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:58707)
0929 14:48:04.126859 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:04.126877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:04.127052 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:04.127359 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:04.127362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:04.127644 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:04.127780 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.127786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.128706 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.128711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.129294 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.129302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.129770 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.129774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.129875 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.130539 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:04.130551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:04.133171 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":108,"threads_started":1}
W20260929 14:48:04.133539 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:04.126563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58707 (local address 127.0.98.62:39383)
0929 14:48:04.126791 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:04.126797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:04.126975 (+ 178us) server_negotiation.cc:407] Connection header received
0929 14:48:04.127167 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:04.127172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:04.127234 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:04.127330 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:04.127904 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.128592 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.129417 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.129647 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.129760 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.133392 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58707: BlockingRecv error: recv got EOF from 127.0.98.2:58707 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:04.338757 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:04.332447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:57495)
0929 14:48:04.332761 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:04.332778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:04.332917 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:04.333294 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:04.333296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:04.333482 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:04.333596 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.333602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.334537 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.334540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.335170 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.335178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.335677 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.335681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.335759 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.336389 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:04.336399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:04.338620 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260929 14:48:04.339128 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:04.332537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57495 (local address 127.0.98.62:39383)
0929 14:48:04.332740 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:04.332745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:04.332875 (+ 130us) server_negotiation.cc:407] Connection header received
0929 14:48:04.333046 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:04.333049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:04.333120 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:04.333242 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:04.333689 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.334421 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.335313 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.335558 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.335677 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.338969 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57495: BlockingRecv error: recv got EOF from 127.0.98.3:57495 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:04 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:04.905946 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:04.898970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:59927)
0929 14:48:04.899337 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:04.899354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:04.899509 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:04.899889 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:04.899891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:04.900111 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:04.900324 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.900331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.901140 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.901143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.901953 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:04.901964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.902811 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.902816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:04.902909 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.903632 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:04.903644 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:04.905776 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
W20260929 14:48:04.906132 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:04.899126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59927 (local address 127.0.98.62:39383)
0929 14:48:04.899387 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:04.899391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:04.899469 (+ 78us) server_negotiation.cc:407] Connection header received
0929 14:48:04.899683 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:04.899686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:04.899741 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:04.899863 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:04.900448 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.901007 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.902105 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:04.902493 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:04.902645 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:04.905934 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59927: BlockingRecv error: recv got EOF from 127.0.98.1:59927 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:05.141088 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:05.134507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:37107)
0929 14:48:05.134862 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:05.134878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:05.135022 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:05.135328 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:05.135331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:05.135557 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:05.135680 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.135686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.136551 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.136555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.137173 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.137181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.137720 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.137723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.137797 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.138414 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:05.138426 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:05.140880 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":80,"threads_started":1}
W20260929 14:48:05.141203 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:05.134852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37107 (local address 127.0.98.62:39383)
0929 14:48:05.135003 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:05.135006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:05.135023 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:48:05.135131 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:05.135133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:05.135203 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:05.135282 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:05.135795 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.136438 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.137325 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.137556 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.137652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.141050 (+ 3398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37107: BlockingRecv error: recv got EOF from 127.0.98.2:37107 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:05.346289 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:05.339806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:51841)
0929 14:48:05.340302 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:05.340324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:05.340502 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:05.340814 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:05.340817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:05.341046 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:05.341142 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.341148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.341985 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.341991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.342599 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.342606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.343124 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.343128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.343217 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.343861 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:05.343873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:05.346108 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":171,"threads_started":1}
W20260929 14:48:05.346410 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:05.340080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51841 (local address 127.0.98.62:39383)
0929 14:48:05.340302 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:05.340309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:05.340425 (+ 116us) server_negotiation.cc:407] Connection header received
0929 14:48:05.340619 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:05.340624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:05.340683 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:05.340788 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:05.341289 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.341866 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.342737 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.342993 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.343111 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.346255 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51841: BlockingRecv error: recv got EOF from 127.0.98.3:51841 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:05 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:05.913311 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:05.906948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:33225)
0929 14:48:05.907297 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:05.907313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:05.907424 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:05.907786 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:05.907789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:05.907993 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:05.908146 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.908153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.909098 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.909100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.909773 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:05.909782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.910339 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.910342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:05.910410 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.910988 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:05.910999 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:05.913159 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
W20260929 14:48:05.913591 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:05.907023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33225 (local address 127.0.98.62:39383)
0929 14:48:05.907369 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:05.907373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:05.907389 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:48:05.907533 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:05.907536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:05.907602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:05.907693 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:05.908368 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.908980 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.909953 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:05.910211 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:05.910305 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:05.913423 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33225: BlockingRecv error: recv got EOF from 127.0.98.1:33225 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:06.148576 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:06.142306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:56259)
0929 14:48:06.142658 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:06.142673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:06.142821 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:06.143099 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:06.143102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:06.143314 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:06.143418 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.143424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.144213 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.144216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.144793 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.144800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.145251 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.145255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.145342 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.146059 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:06.146076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:06.148434 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
W20260929 14:48:06.148850 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:06.142397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56259 (local address 127.0.98.62:39383)
0929 14:48:06.142595 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:06.142600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:06.142752 (+ 152us) server_negotiation.cc:407] Connection header received
0929 14:48:06.142936 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:06.142939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:06.142991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:06.143077 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:06.143532 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.144077 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.144914 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.145136 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.145244 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.148711 (+ 3467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56259: BlockingRecv error: recv got EOF from 127.0.98.2:56259 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:06.353471 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:06.347252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:60497)
0929 14:48:06.347553 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:06.347567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:06.347682 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:06.347979 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:06.347981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:06.348163 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:06.348345 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.348351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.349175 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.349177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.349750 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.349757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.350305 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.350309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.350384 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.351012 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:06.351023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:06.353257 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
W20260929 14:48:06.353724 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:06.347363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60497 (local address 127.0.98.62:39383)
0929 14:48:06.347560 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:06.347564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:06.347660 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:48:06.347784 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:06.347787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:06.347846 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:06.347941 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:06.348468 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.349066 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.349892 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.350138 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.350250 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.353541 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60497: BlockingRecv error: recv got EOF from 127.0.98.3:60497 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:06 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:06.926998 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:06.914207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:44705)
0929 14:48:06.914620 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:06.914637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:06.914746 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:06.915069 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:06.915071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:06.915259 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:06.915375 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.915381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.916276 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.916279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.916939 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:06.916950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.917410 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.917412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:06.917473 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.924737 (+ 7264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:06.924751 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:06.926873 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
W20260929 14:48:06.927270 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:06.914385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44705 (local address 127.0.98.62:39383)
0929 14:48:06.914666 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:06.914670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:06.914693 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:48:06.914865 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:06.914868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:06.914922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:06.915031 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:06.915492 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.916099 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.917062 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:06.917293 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:06.917396 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:06.927078 (+ 9682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44705: BlockingRecv error: recv got EOF from 127.0.98.1:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":69,"threads_started":1}
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:07.157670 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:07.149711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:52583)
0929 14:48:07.150075 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:07.150091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:07.150214 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:07.150710 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:07.150713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:07.150965 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:07.151087 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.151093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.152081 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.152083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.153244 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.153255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.153893 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.153898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.154020 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.154754 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:07.154767 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:07.157490 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
W20260929 14:48:07.157967 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:07.149913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52583 (local address 127.0.98.62:39383)
0929 14:48:07.150180 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:07.150185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:07.150206 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:48:07.150364 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:07.150367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:07.150471 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:07.150564 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:07.151270 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.151903 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.153491 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.153735 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.153865 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.157775 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52583: BlockingRecv error: recv got EOF from 127.0.98.2:52583 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:07.361629 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:07.354779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:39165)
0929 14:48:07.355051 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:07.355071 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:07.355164 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:07.355512 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:07.355515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:07.355760 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:07.355883 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.355889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.356928 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.356935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.357546 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.357557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.358309 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.358313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.358408 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.359181 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:07.359194 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:07.361471 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
W20260929 14:48:07.361936 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:07.354932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39165 (local address 127.0.98.62:39383)
0929 14:48:07.355126 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:07.355130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:07.355149 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:48:07.355273 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:07.355276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:07.355335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:07.355430 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:07.356026 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.356711 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.357715 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.358047 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.358187 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.361638 (+ 3451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39165: BlockingRecv error: recv got EOF from 127.0.98.3:39165 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:07 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:07.934155 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:07.928110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:57745)
0929 14:48:07.928538 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:07.928554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:07.928674 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:07.928927 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:07.928929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:07.929155 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:07.929279 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.929285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.930109 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.930111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.930741 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:07.930749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.931214 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.931216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:07.931278 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.931861 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:07.931902 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:07.933997 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":117,"threads_started":1}
W20260929 14:48:07.934326 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:07.928178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57745 (local address 127.0.98.62:39383)
0929 14:48:07.928559 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:07.928563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:07.928631 (+ 68us) server_negotiation.cc:407] Connection header received
0929 14:48:07.928773 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:07.928775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:07.928822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:07.928914 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:07.929435 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.929994 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.930871 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:07.931098 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:07.931186 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:07.934156 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57745: BlockingRecv error: recv got EOF from 127.0.98.1:57745 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":139,"threads_started":1}
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:08.166095 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:08.158898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:34147)
0929 14:48:08.159387 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:08.159407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:08.159562 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:08.160011 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:08.160017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:08.160326 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:08.160510 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.160518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.161566 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.161573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.162238 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.162247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.162753 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.162757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.162834 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.163488 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:08.163501 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:08.165906 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":177,"threads_started":1}
W20260929 14:48:08.166249 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:08.159185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34147 (local address 127.0.98.62:39383)
0929 14:48:08.159362 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:08.159367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:08.159505 (+ 138us) server_negotiation.cc:407] Connection header received
0929 14:48:08.159725 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:08.159729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:08.159789 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:08.159866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:08.160671 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.161438 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.162372 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.162629 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.162750 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.166110 (+ 3360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34147: BlockingRecv error: recv got EOF from 127.0.98.2:34147 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:08.369388 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:08.362814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:38433)
0929 14:48:08.363210 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:08.363227 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:08.363364 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:08.363717 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:08.363720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:08.363937 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:08.364085 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.364091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.365017 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.365023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.365862 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.365873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.366441 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.366444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.366513 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.367094 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:08.367106 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:08.369237 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
W20260929 14:48:08.369539 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:08.363003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38433 (local address 127.0.98.62:39383)
0929 14:48:08.363204 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:08.363209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:08.363333 (+ 124us) server_negotiation.cc:407] Connection header received
0929 14:48:08.363514 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:08.363517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:08.363577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:08.363666 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:08.364229 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.364899 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.365988 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.366289 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.366429 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.369403 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38433: BlockingRecv error: recv got EOF from 127.0.98.3:38433 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:08 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:08.941571 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:08.935463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:41207)
0929 14:48:08.935765 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:08.935778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:08.935881 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:08.936437 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:08.936443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:08.936654 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:08.936838 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.936848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.937908 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.937910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.938510 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:08.938518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.939014 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.939016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:08.939078 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.939651 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:08.939662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:08.941452 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
W20260929 14:48:08.941877 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:08.935634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41207 (local address 127.0.98.62:39383)
0929 14:48:08.936043 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:08.936047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:08.936066 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:48:08.936122 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:08.936124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:08.936235 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:08.936382 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:08.937013 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.937797 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.938661 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:08.938897 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:08.939001 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:08.941664 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41207: BlockingRecv error: recv got EOF from 127.0.98.1:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:09.173668 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:09.167145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:44847)
0929 14:48:09.167550 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:09.167572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:09.167737 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:09.168024 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:09.168027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:09.168286 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:09.168382 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.168387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.169152 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.169157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.169708 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.169715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.170233 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.170236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.170322 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.171051 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:09.171066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:09.173472 (+ 2406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":118,"threads_started":1}
W20260929 14:48:09.173936 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:09.167327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44847 (local address 127.0.98.62:39383)
0929 14:48:09.167536 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:09.167541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:09.167669 (+ 128us) server_negotiation.cc:407] Connection header received
0929 14:48:09.167842 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:09.167846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:09.167901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:09.167995 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:09.168497 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.169039 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.169829 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.170119 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.170214 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.173752 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44847: BlockingRecv error: recv got EOF from 127.0.98.2:44847 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:09.376721 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:09.370608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:42099)
0929 14:48:09.371031 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:09.371048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:09.371217 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:09.371756 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:09.371759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:09.371992 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:09.372089 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.372096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.372993 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.372999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.373567 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.373575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.374029 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.374032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.374114 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.374722 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:09.374735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:09.376540 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":118,"threads_started":1}
W20260929 14:48:09.376793 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:09.370756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42099 (local address 127.0.98.62:39383)
0929 14:48:09.370994 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:09.370999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:09.371146 (+ 147us) server_negotiation.cc:407] Connection header received
0929 14:48:09.371337 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:09.371340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:09.371404 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:09.371737 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:09.372293 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.372888 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.373693 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.373912 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.374019 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.376675 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42099: BlockingRecv error: recv got EOF from 127.0.98.3:42099 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:09 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:09.948884 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:09.942654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:57975)
0929 14:48:09.942997 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:09.943011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:09.943105 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:09.943422 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:09.943425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:09.943606 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:09.943712 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.943718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.944581 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.944583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.945166 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:09.945174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.945600 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.945603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:09.945671 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.946312 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:09.946323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:09.948685 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
W20260929 14:48:09.949120 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:09.942750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57975 (local address 127.0.98.62:39383)
0929 14:48:09.943088 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:09.943092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:09.943110 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:48:09.943218 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:09.943220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:09.943273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:09.943375 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:09.943836 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.944470 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.945273 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:09.945494 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:09.945574 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:09.948921 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57975: BlockingRecv error: recv got EOF from 127.0.98.1:57975 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:10.181181 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:10.174713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:44079)
0929 14:48:10.175128 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:10.175143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:10.175276 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:10.175581 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:10.175584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:10.175805 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:10.175928 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.175934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.176950 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.176953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.177574 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.177583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.178084 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.178086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.178149 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.178782 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:10.178794 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:10.181039 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":144,"threads_started":1}
W20260929 14:48:10.181402 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:10.174752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44079 (local address 127.0.98.62:39383)
0929 14:48:10.175008 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:10.175019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:10.175265 (+ 246us) server_negotiation.cc:407] Connection header received
0929 14:48:10.175359 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:10.175361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:10.175424 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:10.175514 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:10.176080 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.176826 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.177719 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.177937 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.178033 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.181225 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44079: BlockingRecv error: recv got EOF from 127.0.98.2:44079 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:10.384696 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:10.377936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:51455)
0929 14:48:10.378493 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:10.378513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:10.378693 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:10.379074 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:10.379077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:10.379358 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:10.379517 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.379524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.380626 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.380629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.381251 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.381259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.381718 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.381721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.381790 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.382474 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:10.382487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:10.384554 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":124,"threads_started":1}
W20260929 14:48:10.385003 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:10.378043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51455 (local address 127.0.98.62:39383)
0929 14:48:10.378289 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:10.378295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:10.378764 (+ 469us) server_negotiation.cc:407] Connection header received
0929 14:48:10.378825 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:10.378828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:10.378892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:10.379017 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:10.379718 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.380493 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.381381 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.381597 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.381688 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.384717 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51455: BlockingRecv error: recv got EOF from 127.0.98.3:51455 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:10 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:10.956390 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:10.949854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:60227)
0929 14:48:10.950198 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:10.950214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:10.950328 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:10.950745 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:10.950749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:10.951005 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:10.951165 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.951173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.952101 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.952104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.952856 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:10.952866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.953390 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.953393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:10.953466 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.954006 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:10.954047 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:10.956219 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
W20260929 14:48:10.956553 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:10.950020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60227 (local address 127.0.98.62:39383)
0929 14:48:10.950313 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:10.950317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:10.950333 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:48:10.950499 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:10.950502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:10.950557 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:10.950677 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:10.951289 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.951952 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.953022 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:10.953253 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:10.953339 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:10.956384 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60227: BlockingRecv error: recv got EOF from 127.0.98.1:60227 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":131,"threads_started":1}
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:11.189083 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:11.182533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:43603)
0929 14:48:11.182844 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:11.182858 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:11.183006 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:11.183473 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:11.183476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:11.183691 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:11.183861 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.183869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.185151 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.185154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.185783 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.185792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.186315 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.186318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.186408 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.187063 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:11.187075 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:11.188954 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
W20260929 14:48:11.189350 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:11.182674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43603 (local address 127.0.98.62:39383)
0929 14:48:11.182946 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:11.182950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:11.182965 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:48:11.183206 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:11.183211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:11.183294 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:11.183397 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:11.184154 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.185009 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.185909 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.186177 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.186268 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.189186 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43603: BlockingRecv error: recv got EOF from 127.0.98.2:43603 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:11.392143 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:11.385635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:40183)
0929 14:48:11.385968 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:11.385984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:11.386163 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:11.386441 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:11.386443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:11.386660 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:11.386790 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.386796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.387657 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.387661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.388423 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.388435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.389034 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.389038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.389138 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.389812 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:11.389827 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:11.391995 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":73,"threads_started":1}
W20260929 14:48:11.392392 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:11.385826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40183 (local address 127.0.98.62:39383)
0929 14:48:11.385960 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:11.385964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:11.386114 (+ 150us) server_negotiation.cc:407] Connection header received
0929 14:48:11.386257 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:11.386259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:11.386318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:11.386388 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:11.386935 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.387486 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.388570 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.388859 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.388973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.392211 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40183: BlockingRecv error: recv got EOF from 127.0.98.3:40183 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:11 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:11.969424 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:11.957292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:58559)
0929 14:48:11.957575 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:11.957589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:11.957695 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:11.957960 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:11.957962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:11.958152 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:11.958263 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.958269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.959080 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.959082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.959684 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:11.959692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.960115 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.960118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:11.960212 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.967038 (+ 6826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:11.967051 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:11.969293 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":107,"threads_started":1}
W20260929 14:48:11.969682 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:11.957370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58559 (local address 127.0.98.62:39383)
0929 14:48:11.957672 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:11.957676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:11.957690 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:48:11.957774 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:11.957776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:11.957829 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:11.957918 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:11.958372 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.958960 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.959790 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:11.960014 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:11.960091 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:11.969476 (+ 9385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58559: BlockingRecv error: recv got EOF from 127.0.98.1:58559 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:12.195889 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:12.190047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:48453)
0929 14:48:12.190439 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:12.190455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:12.190569 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:12.190841 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:12.190843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:12.191035 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:12.191135 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.191140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.191966 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.191968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.192589 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.192596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.193059 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.193062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.193121 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.193722 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:12.193732 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:12.195772 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":157,"threads_started":1}
W20260929 14:48:12.196161 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:12.190124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48453 (local address 127.0.98.62:39383)
0929 14:48:12.190305 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:12.190310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:12.190579 (+ 269us) server_negotiation.cc:407] Connection header received
0929 14:48:12.190662 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:12.190664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:12.190723 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:12.190797 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:12.191265 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.191866 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.192703 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.192932 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.193041 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.195975 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48453: BlockingRecv error: recv got EOF from 127.0.98.2:48453 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:12.399595 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:12.393142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:40477)
0929 14:48:12.393528 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:12.393546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:12.393706 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:12.394067 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:12.394070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:12.394410 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:12.394567 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.394595 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.395387 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.395390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.396058 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.396067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.396566 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.396569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.396635 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.397299 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:12.397310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:12.399456 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
W20260929 14:48:12.399864 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:12.393213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40477 (local address 127.0.98.62:39383)
0929 14:48:12.393419 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:12.393424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:12.393640 (+ 216us) server_negotiation.cc:407] Connection header received
0929 14:48:12.393828 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:12.393831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:12.393904 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:12.394000 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:12.394683 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.395279 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.396211 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.396451 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.396536 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.399708 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40477: BlockingRecv error: recv got EOF from 127.0.98.3:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:12 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:12.977423 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:12.970383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:47897)
0929 14:48:12.970824 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:12.970842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:12.970984 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:12.971423 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:12.971429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:12.971695 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:12.971823 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.971829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.972677 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.972679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.973289 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:12.973297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.973771 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.973774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:12.973842 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.974682 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:12.974694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:12.977241 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
W20260929 14:48:12.977483 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:12.970487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47897 (local address 127.0.98.62:39383)
0929 14:48:12.970750 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:12.970763 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:12.970947 (+ 184us) server_negotiation.cc:407] Connection header received
0929 14:48:12.971078 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:12.971081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:12.971167 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:12.971313 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:12.971940 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.972564 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.973395 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:12.973661 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:12.973743 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:12.977360 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47897: BlockingRecv error: recv got EOF from 127.0.98.1:47897 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:13.203022 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:13.196791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:43263)
0929 14:48:13.197110 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:13.197125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:13.197229 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:13.197509 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:13.197511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:13.197695 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:13.197794 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.197799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.198882 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.198884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.199459 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.199467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.199908 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.199911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.199969 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.200607 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:13.200619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:13.202862 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":91,"threads_started":1}
W20260929 14:48:13.203217 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:13.196903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43263 (local address 127.0.98.62:39383)
0929 14:48:13.197129 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:13.197133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:13.197187 (+ 54us) server_negotiation.cc:407] Connection header received
0929 14:48:13.197351 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:13.197354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:13.197408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:13.197482 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:13.197971 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.198772 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.199574 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.199800 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.199884 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.203030 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43263: BlockingRecv error: recv got EOF from 127.0.98.2:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:13.408109 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:13.400605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:50479)
0929 14:48:13.400938 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:13.400991 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:13.401120 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:13.401510 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:13.401512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:13.401733 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:13.401860 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.401867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.402889 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.402891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.404024 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.404042 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.404720 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.404727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.404852 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.405588 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:13.405630 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:13.407918 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
W20260929 14:48:13.408298 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:13.400790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50479 (local address 127.0.98.62:39383)
0929 14:48:13.401013 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:13.401018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:13.401092 (+ 74us) server_negotiation.cc:407] Connection header received
0929 14:48:13.401264 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:13.401267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:13.401328 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:13.401465 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:13.402018 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.402744 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.404204 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.404538 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.404664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.408095 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50479: BlockingRecv error: recv got EOF from 127.0.98.3:50479 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:13 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:13.984787 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:13.978422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:42967)
0929 14:48:13.978830 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:13.978878 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:13.979039 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:13.979385 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:13.979387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:13.979609 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:13.979734 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.979739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.980573 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.980575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.981203 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:13.981211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.981666 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.981669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:13.981739 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.982339 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:13.982350 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:13.984587 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":124,"threads_started":1}
W20260929 14:48:13.984822 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:13.978587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42967 (local address 127.0.98.62:39383)
0929 14:48:13.978845 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:13.978848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:13.978995 (+ 147us) server_negotiation.cc:407] Connection header received
0929 14:48:13.979138 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:13.979140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:13.979189 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:13.979298 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:13.979862 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.980451 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.981331 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:13.981551 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:13.981634 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:13.984704 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42967: BlockingRecv error: recv got EOF from 127.0.98.1:42967 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":61,"threads_started":1}
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:14.210412 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:14.203864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:44853)
0929 14:48:14.204256 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:14.204272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:14.204435 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:14.204758 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:14.204761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:14.205015 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:14.205148 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.205154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.205980 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.205986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.206585 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.206593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.207091 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.207095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.207170 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.207821 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:14.207834 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:14.210194 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
W20260929 14:48:14.210465 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:14.203929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44853 (local address 127.0.98.62:39383)
0929 14:48:14.204136 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:14.204141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:14.204362 (+ 221us) server_negotiation.cc:407] Connection header received
0929 14:48:14.204576 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:14.204579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:14.204640 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:14.204732 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:14.205272 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.205867 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.206714 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.206967 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.207084 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.210349 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44853: BlockingRecv error: recv got EOF from 127.0.98.2:44853 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:14.415912 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:14.409552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:45073)
0929 14:48:14.409925 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:14.409940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:14.410113 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:14.410456 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:14.410459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:14.410678 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:14.410769 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.410775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.411566 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.411571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.412413 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.412422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.413082 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.413085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.413144 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.413910 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:14.413921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:14.415768 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
W20260929 14:48:14.416232 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:14.409655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45073 (local address 127.0.98.62:39383)
0929 14:48:14.409875 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:14.409881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:14.410044 (+ 163us) server_negotiation.cc:407] Connection header received
0929 14:48:14.410259 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:14.410268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:14.410335 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:14.410434 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:14.410924 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.411456 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.412539 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.412841 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.412912 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.416048 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45073: BlockingRecv error: recv got EOF from 127.0.98.3:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:14 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:14.993718 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:14.986085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:59881)
0929 14:48:14.986433 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:14.986448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:14.986600 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:14.986950 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:14.986953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:14.987183 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:14.987315 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.987321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.988232 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.988236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.989220 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:14.989235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.989770 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.989773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:14.989851 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.990656 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:14.990668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:14.993485 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":120,"threads_started":1}
W20260929 14:48:14.993889 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:14.986148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59881 (local address 127.0.98.62:39383)
0929 14:48:14.986446 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:14.986450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:14.986561 (+ 111us) server_negotiation.cc:407] Connection header received
0929 14:48:14.986698 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:14.986701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:14.986748 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:14.986860 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:14.987479 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.988054 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.989380 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:14.989620 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:14.989720 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:14.993679 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59881: BlockingRecv error: recv got EOF from 127.0.98.1:59881 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":127,"threads_started":1}
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:15.218637 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:15.211467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:50063)
0929 14:48:15.211878 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:15.211899 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:15.212054 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:15.212532 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:15.212537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:15.212783 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:15.212913 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.212920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.213855 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.213859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.214475 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.214486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.215224 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.215228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.215329 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:15.216159 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:15.216249 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:15.218442 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":114,"threads_started":1}
W20260929 14:48:15.218935 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:15.211714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50063 (local address 127.0.98.62:39383)
0929 14:48:15.211853 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:15.211857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:15.212017 (+ 160us) server_negotiation.cc:407] Connection header received
0929 14:48:15.212249 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:15.212252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:15.212320 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:15.212424 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:15.213109 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.213643 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.214796 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.215021 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.215126 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:15.218783 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50063: BlockingRecv error: recv got EOF from 127.0.98.2:50063 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:15.425666 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:15.417439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:59895)
0929 14:48:15.417834 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:15.417855 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:15.418186 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:15.418745 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:15.418748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:15.419285 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:15.419425 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.419434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.420331 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.420339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.420897 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.420905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.421381 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.421384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.421466 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:15.422231 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:15.422243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:15.425520 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
W20260929 14:48:15.426102 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:15.417701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59895 (local address 127.0.98.62:39383)
0929 14:48:15.417858 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:15.417863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:15.418011 (+ 148us) server_negotiation.cc:407] Connection header received
0929 14:48:15.418410 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:15.418413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:15.418519 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:15.418637 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:15.419600 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.420121 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.421026 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.421259 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.421373 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:15.425924 (+ 4551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59895: BlockingRecv error: recv got EOF from 127.0.98.3:59895 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:15 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:16.003206 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:15.994962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:37167)
0929 14:48:15.995483 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:15.995506 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:15.995673 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:15.996072 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:15.996075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:15.996588 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:15.996716 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.996726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.997850 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.997854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.998537 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:15.998549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.999150 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.999154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:15.999249 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.000044 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:16.000059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:16.003074 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":113,"threads_started":1}
W20260929 14:48:16.003559 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:15.995304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37167 (local address 127.0.98.62:39383)
0929 14:48:15.995714 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:15.995718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:15.995737 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:48:15.995796 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:15.995799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:15.995849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:15.995956 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:15.996944 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.997719 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.998667 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:15.998997 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:15.999120 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.003318 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37167: BlockingRecv error: recv got EOF from 127.0.98.1:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":117,"threads_started":1}
Sep 29 14:48:16 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:16 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:16.228564 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:16.220087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:55545)
0929 14:48:16.220530 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:16.220548 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:16.220695 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:16.220981 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:16.220984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:16.221266 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:16.221390 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:16.221397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.222861 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.222864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:16.223497 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:16.223506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.224852 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.224855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:16.224935 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.226013 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:16.226030 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:16.228312 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":182,"threads_started":1}
W20260929 14:48:16.228694 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:16.220249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55545 (local address 127.0.98.62:39383)
0929 14:48:16.220450 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:16.220458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:16.220638 (+ 180us) server_negotiation.cc:407] Connection header received
0929 14:48:16.220806 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:16.220809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:16.220867 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:16.220961 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:16.221595 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.222683 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.223961 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.224664 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.224791 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.228555 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55545: BlockingRecv error: recv got EOF from 127.0.98.2:55545 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 14:48:16 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:16 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:16.434453 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:16.427010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:54271)
0929 14:48:16.427397 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:16.427419 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:16.427572 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:16.427958 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:16.427961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:16.428229 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:16.428378 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:16.428384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.429458 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.429461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:16.430080 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:16.430089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.430852 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.430855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:16.430933 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.431621 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:16.431635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:16.434269 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
W20260929 14:48:16.434926 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:16.427052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54271 (local address 127.0.98.62:39383)
0929 14:48:16.427244 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:16.427248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:16.427553 (+ 305us) server_negotiation.cc:407] Connection header received
0929 14:48:16.427717 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:16.427721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:16.427787 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:16.427883 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:16.428665 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.429229 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.430259 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:16.430649 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:16.430755 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:16.434737 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54271: BlockingRecv error: recv got EOF from 127.0.98.3:54271 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:17.015985 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:17.004331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:55425)
0929 14:48:17.004703 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:17.004721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:17.004885 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:17.005331 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:17.005333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:17.005567 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:17.005691 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.005697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.006560 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.006563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.007188 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.007196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.007706 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.007710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.007789 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.013216 (+ 5427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:17.013239 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:17.015830 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":94,"threads_started":1}
W20260929 14:48:17.016180 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:17.004349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55425 (local address 127.0.98.62:39383)
0929 14:48:17.004657 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:17.004661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:17.004827 (+ 166us) server_negotiation.cc:407] Connection header received
0929 14:48:17.004976 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:17.004978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:17.005024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:17.005113 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:17.005836 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.006437 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.007302 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.007557 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.007665 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.016027 (+ 8362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55425: BlockingRecv error: recv got EOF from 127.0.98.1:55425 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":61,"threads_started":1}
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:17.235769 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:17.229804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:46017)
0929 14:48:17.230109 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:17.230123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:17.230229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:17.230543 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:17.230546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:17.230740 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:17.230843 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.230849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.231695 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.231697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.232316 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.232325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.232849 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.232851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.232913 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.233516 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:17.233528 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:17.235599 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260929 14:48:17.235984 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:17.229914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46017 (local address 127.0.98.62:39383)
0929 14:48:17.230083 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:17.230087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:17.230197 (+ 110us) server_negotiation.cc:407] Connection header received
0929 14:48:17.230307 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:17.230309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:17.230385 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:17.230465 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:17.230979 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.231585 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.232454 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.232721 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.232832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.235801 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46017: BlockingRecv error: recv got EOF from 127.0.98.2:46017 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:17 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:17.442031 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:17.435498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:42981)
0929 14:48:17.435833 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:17.435849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:17.435976 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:17.436467 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:17.436470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:17.436683 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:17.436808 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.436814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.437622 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.437624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.438249 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:17.438257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.438740 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.438742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:17.438808 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.439412 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:17.439431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:17.441818 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
W20260929 14:48:17.442124 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:17.435661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42981 (local address 127.0.98.62:39383)
0929 14:48:17.435854 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:17.435857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:17.435949 (+ 92us) server_negotiation.cc:407] Connection header received
0929 14:48:17.436162 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:17.436200 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:17.436276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:17.436376 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:17.436952 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.437509 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.438381 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:17.438617 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:17.438715 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:17.441989 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42981: BlockingRecv error: recv got EOF from 127.0.98.3:42981 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 14:48:18 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:18 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:18.023511 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:18.017118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:60729)
0929 14:48:18.017498 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:18.017518 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:18.017637 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:18.017932 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:18.017935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:18.018152 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:18.018275 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.018281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.019058 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.019061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.019742 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.019751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.020276 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.020279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.020358 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.020963 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:18.020979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:18.023336 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
W20260929 14:48:18.023737 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:18.017243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60729 (local address 127.0.98.62:39383)
0929 14:48:18.017560 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:18.017565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:18.017584 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:48:18.017747 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:18.017749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:18.017798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:18.017875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:18.018385 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.018923 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.019861 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.020084 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.020162 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.023527 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60729: BlockingRecv error: recv got EOF from 127.0.98.1:60729 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
Sep 29 14:48:18 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:48:18 dist-test-slave-5hd1 krb5kdc[8524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:48:18.243284 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:18.236970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:42365)
0929 14:48:18.237386 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:18.237423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:18.237564 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:18.237846 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:18.237849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:18.238124 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:18.238269 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.238276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.239200 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.239203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.239979 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.239988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.240451 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.240454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.240515 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.241101 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:18.241113 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:18.243127 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39383: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
W20260929 14:48:18.243454 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:18.237120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42365 (local address 127.0.98.62:39383)
0929 14:48:18.237313 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:18.237317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:18.237565 (+ 248us) server_negotiation.cc:407] Connection header received
0929 14:48:18.237646 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:18.237649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:18.237700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:18.237813 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:18.238435 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.239086 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.240084 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.240339 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.240422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.243316 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42365: BlockingRecv error: recv got EOF from 127.0.98.2:42365 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 14:48:18.321885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9107
I20260929 14:48:18.328019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9259
I20260929 14:48:18.333673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9391
I20260929 14:48:18.340016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9037
2026-09-29T14:48:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18328 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T14:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:48:18Z Disabled control of system clock
I20260929 14:48:18.373911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43517
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:48:18.485533 9587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:18.485808 9587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:18.485842 9587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:18.490023 9587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:48:18.490082 9587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:18.490098 9587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:18.490115 9587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:48:18.490131 9587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:48:18.494848 9587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:18.495820 9587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:18.497116 9587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:18.505885 9593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:18.506052 9587 server_base.cc:1034] running on GCE node
W20260929 14:48:18.505952 9595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:18.505885 9592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:18.506790 9587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:18.507926 9587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:18.509135 9587 hybrid_clock.cc:648] HybridClock initialized: now 1790693298509086 us; error 67 us; skew 500 ppm
I20260929 14:48:18.511188 9587 webserver.cc:460] Webserver started at http://127.0.98.62:34709/ using document root <none> and password file <none>
I20260929 14:48:18.511786 9587 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:18.511875 9587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:18.512101 9587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:18.514163 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "c67224fe00d748829683fd15625b43e3"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "9e35d6bd62dd9864f0f7be82e2979bea"
server_key_iv: "b2202de78c8f35d6277448d274a80bdb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.514716 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "c67224fe00d748829683fd15625b43e3"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "9e35d6bd62dd9864f0f7be82e2979bea"
server_key_iv: "b2202de78c8f35d6277448d274a80bdb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.518708 9587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 14:48:18.521628 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:18.523140 9587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:48:18.523319 9587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "c67224fe00d748829683fd15625b43e3"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "9e35d6bd62dd9864f0f7be82e2979bea"
server_key_iv: "b2202de78c8f35d6277448d274a80bdb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.523442 9587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:18.538494 9587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:18.539213 9587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:18.539412 9587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:18.546985 9587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43517
I20260929 14:48:18.547004 9653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43517 every 8 connection(s)
I20260929 14:48:18.547964 9587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:48:18.550398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9587
I20260929 14:48:18.550530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:48:18.550908 392 external_mini_cluster.cc:1428] Setting key b41ffc9748f7b24edadd94a8c8bdb1c0
I20260929 14:48:18.552738 9654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:18.561185 9654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3: Bootstrap starting.
I20260929 14:48:18.563800 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:18.552979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.0.98.62:43517)
0929 14:48:18.554024 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:18.554066 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:18.554123 (+ 57us) server_negotiation.cc:407] Connection header received
0929 14:48:18.555148 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:18.555176 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:18.555783 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:18.556278 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:18.556820 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.558366 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.559312 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.560093 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.560316 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.561183 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:18.561190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:18.561210 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:18.561841 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:18.561934 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:18.561963 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:18.562072 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:18.562249 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 14:48:18.562614 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":163,"threads_started":1}
I20260929 14:48:18.564716 9654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:18.565635 9654 log.cc:824] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:18.567772 9654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3: No bootstrap required, opened a new log
I20260929 14:48:18.571566 9654 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260929 14:48:18.571810 9654 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:18.571890 9654 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67224fe00d748829683fd15625b43e3, State: Initialized, Role: FOLLOWER
I20260929 14:48:18.572433 9654 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260929 14:48:18.572569 9654 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:48:18.572654 9654 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:48:18.572757 9654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:18.573778 9654 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260929 14:48:18.574153 9654 leader_election.cc:302] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67224fe00d748829683fd15625b43e3; no voters:
I20260929 14:48:18.574486 9654 leader_election.cc:288] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:48:18.574667 9659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:48:18.574959 9659 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [term 1 LEADER]: Becoming Leader. State: Replica: c67224fe00d748829683fd15625b43e3, State: Running, Role: LEADER
I20260929 14:48:18.575325 9659 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } }
I20260929 14:48:18.575934 9654 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:48:18.577207 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67224fe00d748829683fd15625b43e3. Latest consensus state: current_term: 1 leader_uuid: "c67224fe00d748829683fd15625b43e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } } }
I20260929 14:48:18.577361 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:18.577813 9668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:48:18.578401 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67224fe00d748829683fd15625b43e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67224fe00d748829683fd15625b43e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43517 } } }
I20260929 14:48:18.578539 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:18.580747 9668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:48:18.587502 9668 catalog_manager.cc:1347] Generated new cluster ID: 1137d12e320f4a36b9c461e430d2bc74
I20260929 14:48:18.587580 9668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:48:18.599205 9668 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:48:18.600153 9668 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:48:18.607848 9668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67224fe00d748829683fd15625b43e3: Generated new TSK 0
I20260929 14:48:18.608769 9668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:48:18.619788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 14:48:18.745184 9678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:18.745496 9678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:18.745536 9678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:18.749923 9678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:18.749994 9678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:18.750073 9678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:48:18.755066 9678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:18.756294 9678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:18.757686 9678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:18.767592 9686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:18.767592 9684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:18.767647 9678 server_base.cc:1034] running on GCE node
W20260929 14:48:18.767663 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:18.768380 9678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:18.769018 9678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:18.770185 9678 hybrid_clock.cc:648] HybridClock initialized: now 1790693298770147 us; error 60 us; skew 500 ppm
I20260929 14:48:18.772219 9678 webserver.cc:460] Webserver started at http://127.0.98.1:35435/ using document root <none> and password file <none>
I20260929 14:48:18.772809 9678 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:18.772873 9678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:18.773038 9678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:18.775034 9678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b74685a338240c1b67c74534d0d02f9"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "f44f353ce6ccb8310822f15ba9001f15"
server_key_iv: "04c75ae94aacffb786c207570b6891fb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.775539 9678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b74685a338240c1b67c74534d0d02f9"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "f44f353ce6ccb8310822f15ba9001f15"
server_key_iv: "04c75ae94aacffb786c207570b6891fb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.779853 9678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:48:18.782709 9692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:18.784253 9678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:18.784394 9678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "4b74685a338240c1b67c74534d0d02f9"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "f44f353ce6ccb8310822f15ba9001f15"
server_key_iv: "04c75ae94aacffb786c207570b6891fb"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.784520 9678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:18.798545 9678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:18.799257 9678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:18.799461 9678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:18.800076 9678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:18.801093 9678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:18.801148 9678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:18.801221 9678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:18.801258 9678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:18.811365 9678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32987
I20260929 14:48:18.811483 9805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32987 every 8 connection(s)
I20260929 14:48:18.812398 9678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:48:18.817420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9678
I20260929 14:48:18.817591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:48:18.817965 392 external_mini_cluster.cc:1428] Setting key de651f16cce6921b2208db71832a353f
I20260929 14:48:18.820430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:48:18.823230 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:18.814813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58721 (local address 127.0.98.62:43517)
0929 14:48:18.814958 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:18.814962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:18.816453 (+ 1491us) server_negotiation.cc:407] Connection header received
0929 14:48:18.818025 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:18.818028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:18.818093 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:18.818182 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:18.819236 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.819911 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.821186 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.821418 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.821526 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.822663 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:18.822670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:18.822673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:18.822698 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:18.822718 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:18.822726 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:18.822812 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:18.822975 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 14:48:18.823066 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:48:18.824237 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:18.815322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.1:58721)
0929 14:48:18.816288 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:18.816339 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:18.817800 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:18.818406 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:18.818418 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:18.818528 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:18.819085 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.819099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.820040 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.820044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.821066 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:18.821075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:18.821572 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:18.821576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:18.821739 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:18.822381 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:18.822383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:18.822424 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:18.822492 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:18.822814 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:18.822825 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:18.822833 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:48:18.822965 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 14:48:18.823278 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":157,"threads_started":1}
I20260929 14:48:18.825717 9806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260929 14:48:18.826027 9806 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:18.826718 9806 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260929 14:48:18.828491 9616 ts_manager.cc:194] Registered new tserver with Master: 4b74685a338240c1b67c74534d0d02f9 (127.0.98.1:32987)
I20260929 14:48:18.829646 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58721
W20260929 14:48:18.950199 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:18.950585 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:18.950668 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:18.954949 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:18.955040 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:18.955140 9809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:48:18.960160 9809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:18.961655 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:18.963521 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:18.974721 9817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:18.974762 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:18.974721 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:18.975180 9809 server_base.cc:1034] running on GCE node
I20260929 14:48:18.975718 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:18.976569 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:18.977770 9809 hybrid_clock.cc:648] HybridClock initialized: now 1790693298977752 us; error 35 us; skew 500 ppm
I20260929 14:48:18.979832 9809 webserver.cc:460] Webserver started at http://127.0.98.2:36347/ using document root <none> and password file <none>
I20260929 14:48:18.980536 9809 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:18.980604 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:18.980825 9809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:18.982956 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "1dc4f3fb76344a62a6264a032ee480f7"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "2093b1478084cacb6a6efda41cb6b22b"
server_key_iv: "f52a01292719beef9e222b3111745d6e"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.983532 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1dc4f3fb76344a62a6264a032ee480f7"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "2093b1478084cacb6a6efda41cb6b22b"
server_key_iv: "f52a01292719beef9e222b3111745d6e"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.987977 9809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:48:18.991155 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:18.992826 9809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:18.992962 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "1dc4f3fb76344a62a6264a032ee480f7"
format_stamp: "Formatted at 2026-09-29 14:48:18 on dist-test-slave-5hd1"
server_key: "2093b1478084cacb6a6efda41cb6b22b"
server_key_iv: "f52a01292719beef9e222b3111745d6e"
server_key_version: "encryptionkey@0"
I20260929 14:48:18.993104 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:19.018564 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:19.019317 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:19.019527 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:19.020138 9809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:19.021216 9809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:19.021267 9809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.021332 9809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:19.021370 9809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.031986 9809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38961
I20260929 14:48:19.032048 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38961 every 8 connection(s)
I20260929 14:48:19.033082 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:48:19.038703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9809
I20260929 14:48:19.038869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:48:19.039206 392 external_mini_cluster.cc:1428] Setting key 0ab99b6daaaee0e14044d78e369c9801
I20260929 14:48:19.041651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--builtin_ntp_servers=127.0.98.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:48:19.046073 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.035582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49119 (local address 127.0.98.62:43517)
0929 14:48:19.035797 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:19.035801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:19.037451 (+ 1650us) server_negotiation.cc:407] Connection header received
0929 14:48:19.039327 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:19.039334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:19.039433 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:19.039552 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:19.040957 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.041612 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.043527 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.043942 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.044095 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:19.045395 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:19.045410 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:19.045417 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:19.045439 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:19.045472 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:19.045485 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:19.045576 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:19.045738 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 14:48:19.045828 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 14:48:19.046885 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.036360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.2:49119)
0929 14:48:19.037267 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:19.037320 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:19.038959 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:19.039783 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:19.039790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:19.039885 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:19.040602 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:19.040621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.041772 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.041779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:19.043372 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:19.043383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.044149 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.044153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:19.044412 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:19.045083 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:19.045084 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:19.045128 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:19.045210 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:19.045583 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:19.045589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:19.045594 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:48:19.045693 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 14:48:19.045930 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":156,"threads_started":1}
I20260929 14:48:19.048243 9937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260929 14:48:19.048485 9937 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:19.049008 9937 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260929 14:48:19.050081 9616 ts_manager.cc:194] Registered new tserver with Master: 1dc4f3fb76344a62a6264a032ee480f7 (127.0.98.2:38961)
I20260929 14:48:19.050657 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49119
W20260929 14:48:19.177377 9940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:19.177769 9940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:19.177856 9940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:19.183020 9940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:19.183112 9940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:19.183224 9940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:48:19.189400 9940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:19.191046 9940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:19.192992 9940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:19.205317 9945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.205464 9948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.205317 9946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:19.205996 9940 server_base.cc:1034] running on GCE node
I20260929 14:48:19.206452 9940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:19.207131 9940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:19.208331 9940 hybrid_clock.cc:648] HybridClock initialized: now 1790693299208307 us; error 47 us; skew 500 ppm
I20260929 14:48:19.210476 9940 webserver.cc:460] Webserver started at http://127.0.98.3:45987/ using document root <none> and password file <none>
I20260929 14:48:19.211117 9940 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:19.211220 9940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:19.211460 9940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:19.213631 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "e216733f082c4b60ad047d908fbb0b0f"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "b97db983844b65a1a0bfd2fdb51a6075"
server_key_iv: "36efd6ba84ee2450e29c390cfee4e999"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.214215 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e216733f082c4b60ad047d908fbb0b0f"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "b97db983844b65a1a0bfd2fdb51a6075"
server_key_iv: "36efd6ba84ee2450e29c390cfee4e999"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.218303 9940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 14:48:19.221387 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.223136 9940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:19.223311 9940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "e216733f082c4b60ad047d908fbb0b0f"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "b97db983844b65a1a0bfd2fdb51a6075"
server_key_iv: "36efd6ba84ee2450e29c390cfee4e999"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.223450 9940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:19.258481 9940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:19.259231 9940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:19.259434 9940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:19.260114 9940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:19.261126 9940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:19.261207 9940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.261296 9940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:19.261343 9940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.271548 9940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42257
I20260929 14:48:19.271620 10067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42257 every 8 connection(s)
I20260929 14:48:19.272594 9940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:48:19.280385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 9940
I20260929 14:48:19.280526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:48:19.280859 392 external_mini_cluster.cc:1428] Setting key 935793a9ae614f8b8a95f8d79f304a5f
I20260929 14:48:19.284087 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.275008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57597 (local address 127.0.98.62:43517)
0929 14:48:19.275224 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:19.275230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:19.276876 (+ 1646us) server_negotiation.cc:407] Connection header received
0929 14:48:19.278480 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:19.278483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:19.278571 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:19.278684 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:19.279723 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.280357 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.281801 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.282164 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.282289 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:19.283503 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:19.283507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:19.283510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:19.283524 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:19.283555 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:19.283564 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:19.283629 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:19.283885 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 14:48:19.283950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:48:19.285028 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.275547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43517 (local address 127.0.98.3:57597)
0929 14:48:19.276560 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:19.276648 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:19.278177 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:19.278914 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:19.278922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:19.279014 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:19.279539 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:19.279552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.280508 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.280512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:19.281680 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:19.281691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:19.282346 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:19.282354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:19.282527 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:19.283173 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:19.283175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:19.283244 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:19.283331 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:19.283670 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:19.283678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:19.283683 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:48:19.283824 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 14:48:19.284129 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":150,"threads_started":1}
I20260929 14:48:19.286273 10068 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43517
I20260929 14:48:19.286702 10068 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:19.287488 10068 heartbeater.cc:511] Master 127.0.98.62:43517 requested a full tablet report, sending...
I20260929 14:48:19.288939 9616 ts_manager.cc:194] Registered new tserver with Master: e216733f082c4b60ad047d908fbb0b0f (127.0.98.3:42257)
I20260929 14:48:19.289613 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57597
I20260929 14:48:19.295320 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:48:19.298971 9657 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:19.297769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.0.98.62:43517)
0929 14:48:19.297941 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:19.297947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:19.298191 (+ 244us) server_negotiation.cc:407] Connection header received
0929 14:48:19.298373 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:19.298377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:19.298436 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:19.298524 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:19.298804 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46398: BlockingRecv error: recv got EOF from 127.0.0.1:46398 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:48:19.299185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9678
I20260929 14:48:19.305658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9809
I20260929 14:48:19.311444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9940
I20260929 14:48:19.317941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 9587
2026-09-29T14:48:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (977 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T14:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:48:19Z Disabled control of system clock
I20260929 14:48:19.352458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39945
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:48:19.490761 10079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:19.491135 10079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:19.491182 10079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:19.495879 10079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:48:19.495955 10079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:19.495971 10079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:19.495990 10079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:48:19.496006 10079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:48:19.501093 10079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:19.502473 10079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:19.504142 10079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:19.514639 10087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.514636 10085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.514636 10084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:19.515334 10079 server_base.cc:1034] running on GCE node
I20260929 14:48:19.515987 10079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:19.517275 10079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:19.518496 10079 hybrid_clock.cc:648] HybridClock initialized: now 1790693299518479 us; error 44 us; skew 500 ppm
I20260929 14:48:19.520793 10079 webserver.cc:460] Webserver started at http://127.0.98.62:43521/ using document root <none> and password file <none>
I20260929 14:48:19.521445 10079 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:19.521546 10079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:19.521785 10079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:19.524252 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "a1e5cabe1e9d4fb188fcc435f9129d67"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "3529d554b2216c0c57418c8c59f59118"
server_key_iv: "cc8ae1b1655bba1658f94553918bbc4f"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.524896 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1e5cabe1e9d4fb188fcc435f9129d67"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "3529d554b2216c0c57418c8c59f59118"
server_key_iv: "cc8ae1b1655bba1658f94553918bbc4f"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.529240 10079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:48:19.532231 10093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.533773 10079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:19.533941 10079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "a1e5cabe1e9d4fb188fcc435f9129d67"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "3529d554b2216c0c57418c8c59f59118"
server_key_iv: "cc8ae1b1655bba1658f94553918bbc4f"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.534071 10079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:19.538826 10079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:19.539465 10079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:19.539652 10079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:19.547049 10079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39945
I20260929 14:48:19.547060 10145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39945 every 8 connection(s)
I20260929 14:48:19.548262 10079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:48:19.550354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10079
I20260929 14:48:19.550474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:48:19.550801 392 external_mini_cluster.cc:1428] Setting key 1f03ff7e980b46267d6ba6a673dfbb32
I20260929 14:48:19.552124 10146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:19.559105 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.552775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.98.62:39945)
0929 14:48:19.553681 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:19.553702 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:19.553753 (+ 51us) server_negotiation.cc:407] Connection header received
0929 14:48:19.555404 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:19.555438 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:19.555958 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:19.556442 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:19.557501 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:19.557507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:19.557521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:19.557534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:19.557607 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:19.557633 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:19.557782 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:19.557881 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 14:48:19.558233 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260929 14:48:19.559880 10146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: Bootstrap starting.
I20260929 14:48:19.562211 10146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:19.562952 10146 log.cc:824] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:19.565279 10146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: No bootstrap required, opened a new log
I20260929 14:48:19.568850 10146 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260929 14:48:19.569073 10146 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:19.569114 10146 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e5cabe1e9d4fb188fcc435f9129d67, State: Initialized, Role: FOLLOWER
I20260929 14:48:19.569571 10146 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260929 14:48:19.569694 10146 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:48:19.569738 10146 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:48:19.569860 10146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:19.570749 10146 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260929 14:48:19.571097 10146 leader_election.cc:302] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e5cabe1e9d4fb188fcc435f9129d67; no voters:
I20260929 14:48:19.571419 10146 leader_election.cc:288] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:48:19.571717 10151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:48:19.572427 10151 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [term 1 LEADER]: Becoming Leader. State: Replica: a1e5cabe1e9d4fb188fcc435f9129d67, State: Running, Role: LEADER
I20260929 14:48:19.572813 10146 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:48:19.572881 10151 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } }
I20260929 14:48:19.574892 10153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e5cabe1e9d4fb188fcc435f9129d67. Latest consensus state: current_term: 1 leader_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } }
I20260929 14:48:19.575034 10153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:19.575206 10152 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e5cabe1e9d4fb188fcc435f9129d67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39945 } } }
I20260929 14:48:19.575762 10152 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67 [sys.catalog]: This master's current role is: LEADER
W20260929 14:48:19.578552 10165 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:48:19.578642 10165 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:48:19.578714 10166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:48:19.579578 10166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:48:19.586171 10166 catalog_manager.cc:1347] Generated new cluster ID: c4e502f29ebd4899970951ef61f1ec28
I20260929 14:48:19.586256 10166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:48:19.601415 10166 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:48:19.601809 10166 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:48:19.612419 10166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1e5cabe1e9d4fb188fcc435f9129d67: Generated new TSK 0
I20260929 14:48:19.613474 10166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:48:19.626986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:48:19.760488 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:19.760833 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:19.760911 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:19.765333 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:19.765400 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:19.765465 10170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:48:19.770550 10170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:19.771996 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:19.773846 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:19.785568 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.785553 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:19.785844 10170 server_base.cc:1034] running on GCE node
W20260929 14:48:19.785563 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:19.786455 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:19.787246 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:19.788436 10170 hybrid_clock.cc:648] HybridClock initialized: now 1790693299788399 us; error 60 us; skew 500 ppm
I20260929 14:48:19.790652 10170 webserver.cc:460] Webserver started at http://127.0.98.1:34241/ using document root <none> and password file <none>
I20260929 14:48:19.791404 10170 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:19.791499 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:19.791733 10170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:19.794054 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "b97af67967cf4090a1057270a11c2f21"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "de854476ff7a2192322a3f4e38dc943a"
server_key_iv: "c94f729863097ec701335467c3006d93"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.794623 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b97af67967cf4090a1057270a11c2f21"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "de854476ff7a2192322a3f4e38dc943a"
server_key_iv: "c94f729863097ec701335467c3006d93"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.798993 10170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 14:48:19.801733 10184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.803156 10170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:48:19.803311 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "b97af67967cf4090a1057270a11c2f21"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "de854476ff7a2192322a3f4e38dc943a"
server_key_iv: "c94f729863097ec701335467c3006d93"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.803426 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:19.807969 10170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:19.808643 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:19.808828 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:19.809393 10170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:19.810333 10170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:19.810400 10170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.810465 10170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:19.810504 10170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:19.821142 10170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44217
I20260929 14:48:19.821264 10297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44217 every 8 connection(s)
I20260929 14:48:19.822245 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:48:19.825441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10170
I20260929 14:48:19.825563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:48:19.826021 392 external_mini_cluster.cc:1428] Setting key f4af6e5cd5500bb81800156412f6be10
I20260929 14:48:19.828658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 14:48:19.830892 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.824990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59975 (local address 127.0.98.62:39945)
0929 14:48:19.825255 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:19.825262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:19.826755 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 14:48:19.828580 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:19.828583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:19.828621 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:19.828711 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:19.830287 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:19.830290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:19.830293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:19.830313 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:19.830334 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:19.830342 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:19.830458 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:19.830661 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 14:48:19.830714 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 14:48:19.831980 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:19.825419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.1:59975)
0929 14:48:19.826548 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:19.826607 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:19.828253 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:19.828992 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:19.829005 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:19.829148 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:19.829918 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:19.829922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:19.829992 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:19.830102 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:19.830445 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:19.830457 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:19.830467 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 14:48:19.830705 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 14:48:19.830956 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":170,"threads_started":1}
I20260929 14:48:19.833670 10298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260929 14:48:19.834012 10298 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:19.834533 10298 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260929 14:48:19.836838 10107 ts_manager.cc:194] Registered new tserver with Master: b97af67967cf4090a1057270a11c2f21 (127.0.98.1:44217)
W20260929 14:48:19.959839 10301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:19.960207 10301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:19.960253 10301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:19.964488 10301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:19.964566 10301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:19.964628 10301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:48:19.969807 10301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:19.971194 10301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:19.972950 10301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:19.984436 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.984449 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:19.984529 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:19.984661 10301 server_base.cc:1034] running on GCE node
I20260929 14:48:19.985348 10301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:19.986127 10301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:19.987358 10301 hybrid_clock.cc:648] HybridClock initialized: now 1790693299987307 us; error 69 us; skew 500 ppm
I20260929 14:48:19.989603 10301 webserver.cc:460] Webserver started at http://127.0.98.2:32981/ using document root <none> and password file <none>
I20260929 14:48:19.990293 10301 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:19.990401 10301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:19.990648 10301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:19.993038 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "5acaaae115e74329921d4f5de55ec080"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "8304740a1dcba97b4d93e3e4473312a7"
server_key_iv: "544e63545de09f535af53ad1166ed8c8"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.993638 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5acaaae115e74329921d4f5de55ec080"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "8304740a1dcba97b4d93e3e4473312a7"
server_key_iv: "544e63545de09f535af53ad1166ed8c8"
server_key_version: "encryptionkey@0"
I20260929 14:48:19.998131 10301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:48:20.001269 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.002919 10301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:20.003109 10301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "5acaaae115e74329921d4f5de55ec080"
format_stamp: "Formatted at 2026-09-29 14:48:19 on dist-test-slave-5hd1"
server_key: "8304740a1dcba97b4d93e3e4473312a7"
server_key_iv: "544e63545de09f535af53ad1166ed8c8"
server_key_version: "encryptionkey@0"
I20260929 14:48:20.003243 10301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:20.008455 10301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:20.009137 10301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:20.009322 10301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:20.009887 10301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:20.010864 10301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:20.010931 10301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.010999 10301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:20.011039 10301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.021243 10301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39375
I20260929 14:48:20.021311 10428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39375 every 8 connection(s)
I20260929 14:48:20.022300 10301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:48:20.028319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10301
I20260929 14:48:20.028506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:48:20.028904 392 external_mini_cluster.cc:1428] Setting key a92e5e2037e1835167b9c9ce6d19388d
I20260929 14:48:20.029914 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:20.024743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52729 (local address 127.0.98.62:39945)
0929 14:48:20.024942 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:20.024947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:20.026302 (+ 1355us) server_negotiation.cc:407] Connection header received
0929 14:48:20.027683 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:20.027686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:20.027731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:20.027808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:20.029172 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:20.029177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:20.029181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:20.029199 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:20.029233 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:20.029247 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:20.029400 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:20.029696 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 14:48:20.029755 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:48:20.031033 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:20.025219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.2:52729)
0929 14:48:20.026118 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:20.026168 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:20.027456 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:20.027982 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:20.027989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:20.028081 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:20.028853 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:20.028854 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:20.028895 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:20.028962 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:20.029439 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:20.029454 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:20.029462 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:48:20.029724 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 14:48:20.029967 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":156,"threads_started":1}
I20260929 14:48:20.031728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--builtin_ntp_servers=127.0.98.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 14:48:20.032649 10429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260929 14:48:20.032938 10429 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:20.033414 10429 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260929 14:48:20.034788 10107 ts_manager.cc:194] Registered new tserver with Master: 5acaaae115e74329921d4f5de55ec080 (127.0.98.2:39375)
W20260929 14:48:20.153470 10432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:20.153764 10432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:20.153806 10432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:20.157932 10432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:20.157998 10432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:20.158057 10432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:48:20.162992 10432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:20.164143 10432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:20.165573 10432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:20.176298 10438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:20.176319 10437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:20.176513 10432 server_base.cc:1034] running on GCE node
W20260929 14:48:20.176363 10440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:20.177170 10432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:20.177958 10432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:20.179158 10432 hybrid_clock.cc:648] HybridClock initialized: now 1790693300179121 us; error 50 us; skew 500 ppm
I20260929 14:48:20.181486 10432 webserver.cc:460] Webserver started at http://127.0.98.3:42825/ using document root <none> and password file <none>
I20260929 14:48:20.182190 10432 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:20.182281 10432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:20.182539 10432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:20.185009 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "3c96280fa41a45e8b03dd7d9107c5e56"
format_stamp: "Formatted at 2026-09-29 14:48:20 on dist-test-slave-5hd1"
server_key: "17b781ce364287c8288df4d464db090d"
server_key_iv: "4a249575276491695a68687d36f51607"
server_key_version: "encryptionkey@0"
I20260929 14:48:20.185617 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3c96280fa41a45e8b03dd7d9107c5e56"
format_stamp: "Formatted at 2026-09-29 14:48:20 on dist-test-slave-5hd1"
server_key: "17b781ce364287c8288df4d464db090d"
server_key_iv: "4a249575276491695a68687d36f51607"
server_key_version: "encryptionkey@0"
I20260929 14:48:20.189940 10432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:48:20.193042 10446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.194576 10432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:20.194741 10432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "3c96280fa41a45e8b03dd7d9107c5e56"
format_stamp: "Formatted at 2026-09-29 14:48:20 on dist-test-slave-5hd1"
server_key: "17b781ce364287c8288df4d464db090d"
server_key_iv: "4a249575276491695a68687d36f51607"
server_key_version: "encryptionkey@0"
I20260929 14:48:20.194895 10432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:20.200096 10432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:20.200877 10432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:20.201083 10432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:20.201737 10432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:20.202837 10432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:20.202920 10432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.202994 10432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:20.203044 10432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:20.214906 10432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41549
I20260929 14:48:20.214926 10559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41549 every 8 connection(s)
I20260929 14:48:20.216257 10432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:48:20.219749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10432
I20260929 14:48:20.219898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:48:20.220316 392 external_mini_cluster.cc:1428] Setting key 3d9dabe41c68ade202a7defe4ef12327
I20260929 14:48:20.224414 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:20.218756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55837 (local address 127.0.98.62:39945)
0929 14:48:20.218982 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:20.218988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:20.220349 (+ 1361us) server_negotiation.cc:407] Connection header received
0929 14:48:20.222293 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:20.222296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:20.222341 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:20.222441 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:20.223903 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:20.223907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:20.223909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:48:20.223922 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:20.223945 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:48:20.223953 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:20.224049 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:20.224251 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 14:48:20.224297 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:48:20.225551 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:20.219220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39945 (local address 127.0.98.3:55837)
0929 14:48:20.220103 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:20.220164 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:20.222001 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:20.222695 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:20.222705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:20.222861 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:20.223592 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:48:20.223594 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:20.223642 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:48:20.223726 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:20.224052 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:20.224059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:20.224063 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:48:20.224283 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 14:48:20.224581 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":172,"threads_started":1}
I20260929 14:48:20.226884 10560 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39945
I20260929 14:48:20.227213 10560 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:20.227694 10560 heartbeater.cc:511] Master 127.0.98.62:39945 requested a full tablet report, sending...
I20260929 14:48:20.228930 10107 ts_manager.cc:194] Registered new tserver with Master: 3c96280fa41a45e8b03dd7d9107c5e56 (127.0.98.3:41549)
I20260929 14:48:20.235399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:48:20.239099 10149 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:48:20.237745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.0.98.62:39945)
0929 14:48:20.237885 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:20.237889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:20.238203 (+ 314us) server_negotiation.cc:407] Connection header received
0929 14:48:20.238312 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:20.238315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:20.238387 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:20.238478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:20.238945 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39282: BlockingRecv error: recv got EOF from 127.0.0.1:39282 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:48:20.239322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10170
I20260929 14:48:20.245816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10301
I20260929 14:48:20.252974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10432
I20260929 14:48:20.259815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10079
2026-09-29T14:48:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (940 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:48:20 dist-test-slave-5hd1 krb5kdc[10570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:48:20 dist-test-slave-5hd1 krb5kdc[10570](info): set up 2 sockets
Sep 29 14:48:20 dist-test-slave-5hd1 krb5kdc[10570](info): commencing operation
krb5kdc: starting...
W20260929 14:48:21.325615 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:48:21 dist-test-slave-5hd1 krb5kdc[10570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:48:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:48:21.465667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:21.596416 10583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:21.596877 10583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:21.596961 10583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:21.602794 10583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:48:21.602927 10583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:21.602967 10583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:21.603014 10583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:48:21.603051 10583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:48:21.608980 10583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:21.610178 10583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:21.611241 10583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:21.620641 10591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:21.620641 10589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:21.620786 10588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:21.620836 10583 server_base.cc:1034] running on GCE node
I20260929 14:48:21.621791 10583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:21.623431 10583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:21.624644 10583 hybrid_clock.cc:648] HybridClock initialized: now 1790693301624628 us; error 38 us; skew 500 ppm
Sep 29 14:48:21 dist-test-slave-5hd1 krb5kdc[10570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:21.627943 10583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:48:21.629135 10583 webserver.cc:460] Webserver started at http://127.0.98.62:37807/ using document root <none> and password file <none>
I20260929 14:48:21.629823 10583 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:21.629923 10583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:21.630182 10583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:21.632709 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "c77d1e2229334c44b14a796f02e6eecc"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "12c3619972094e691cf2eac871552a01"
server_key_iv: "a0afc03c809dd6e8f26d9ad688ed01d6"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.633373 10583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "c77d1e2229334c44b14a796f02e6eecc"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "12c3619972094e691cf2eac871552a01"
server_key_iv: "a0afc03c809dd6e8f26d9ad688ed01d6"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.638352 10583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:48:21.641405 10598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:21.643023 10583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:48:21.643186 10583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "c77d1e2229334c44b14a796f02e6eecc"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "12c3619972094e691cf2eac871552a01"
server_key_iv: "a0afc03c809dd6e8f26d9ad688ed01d6"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.643321 10583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:21.656379 10583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:21.659876 10583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:21.660115 10583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:21.668694 10583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42115
I20260929 14:48:21.669024 10650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42115 every 8 connection(s)
I20260929 14:48:21.669958 10583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:48:21.673225 10651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:21.673552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10583
I20260929 14:48:21.673707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:48:21.674043 392 external_mini_cluster.cc:1428] Setting key 38e94bb35823644336d8c0e25b7f002b
I20260929 14:48:21.682089 10651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: Bootstrap starting.
I20260929 14:48:21.685479 10651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: Neither blocks nor log segments found. Creating new log.
Sep 29 14:48:21 dist-test-slave-5hd1 krb5kdc[10570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:21.686295 10651 log.cc:824] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:21.689414 10651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: No bootstrap required, opened a new log
I20260929 14:48:21.693329 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:21.675657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.0.98.62:42115)
0929 14:48:21.676135 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:21.676155 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:21.676255 (+ 100us) server_negotiation.cc:407] Connection header received
0929 14:48:21.677142 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:21.677165 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:21.677627 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:21.678006 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:21.679084 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:21.680528 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:21.682035 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:21.682915 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:21.683106 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:21.686784 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:21.686791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:21.686833 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:21.686887 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:21.688997 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:21.689697 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:21.689705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:21.689712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:21.689886 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:21.690282 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:21.690286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:21.690288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:21.690913 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:21.691133 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:21.691490 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 14:48:21.691892 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":128,"threads_started":1}
I20260929 14:48:21.695004 10651 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260929 14:48:21.695351 10651 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:21.695403 10651 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77d1e2229334c44b14a796f02e6eecc, State: Initialized, Role: FOLLOWER
I20260929 14:48:21.695971 10651 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260929 14:48:21.696175 10651 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:48:21.696312 10651 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:48:21.696434 10651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:21.697619 10651 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260929 14:48:21.698053 10651 leader_election.cc:302] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77d1e2229334c44b14a796f02e6eecc; no voters:
I20260929 14:48:21.698462 10651 leader_election.cc:288] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:48:21.698836 10656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:48:21.699193 10656 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [term 1 LEADER]: Becoming Leader. State: Replica: c77d1e2229334c44b14a796f02e6eecc, State: Running, Role: LEADER
I20260929 14:48:21.699581 10656 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } }
I20260929 14:48:21.700045 10651 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:48:21.701646 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77d1e2229334c44b14a796f02e6eecc. Latest consensus state: current_term: 1 leader_uuid: "c77d1e2229334c44b14a796f02e6eecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260929 14:48:21.701712 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77d1e2229334c44b14a796f02e6eecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77d1e2229334c44b14a796f02e6eecc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42115 } } }
I20260929 14:48:21.701797 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:21.701830 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc [sys.catalog]: This master's current role is: LEADER
I20260929 14:48:21.702431 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:48:21.706036 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:48:21.712100 10666 catalog_manager.cc:1347] Generated new cluster ID: 5abf1d8ecf394ae8a6c69383c17bada9
I20260929 14:48:21.712217 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:48:21.729357 10666 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:48:21.730535 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:48:21.740109 10666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: Generated new TSK 0
I20260929 14:48:21.741256 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:48:21.779852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:21.912532 10677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:21.912837 10677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:21.912885 10677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:21.917073 10677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:21.917131 10677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:21.917194 10677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:48:21.922961 10677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:21.924366 10677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:21.925587 10677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:21.937239 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:21.937331 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:21.937275 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:21.937940 10677 server_base.cc:1034] running on GCE node
I20260929 14:48:21.938467 10677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:21.939330 10677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:21.940557 10677 hybrid_clock.cc:648] HybridClock initialized: now 1790693301940538 us; error 53 us; skew 500 ppm
Sep 29 14:48:21 dist-test-slave-5hd1 krb5kdc[10570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:21.944347 10677 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:48:21.945482 10677 webserver.cc:460] Webserver started at http://127.0.98.1:37075/ using document root <none> and password file <none>
I20260929 14:48:21.946154 10677 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:21.946255 10677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:21.946509 10677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:21.948873 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd5de2fdb5d14550ad95369eb7f899f4"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "0277380b58433c0a2bb8f7cdf8197afe"
server_key_iv: "6fd9b707f27ec0545a23b0646a8c3ee7"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.949474 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd5de2fdb5d14550ad95369eb7f899f4"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "0277380b58433c0a2bb8f7cdf8197afe"
server_key_iv: "6fd9b707f27ec0545a23b0646a8c3ee7"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.953742 10677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 14:48:21.956526 10692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:21.957989 10677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:21.958125 10677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "cd5de2fdb5d14550ad95369eb7f899f4"
format_stamp: "Formatted at 2026-09-29 14:48:21 on dist-test-slave-5hd1"
server_key: "0277380b58433c0a2bb8f7cdf8197afe"
server_key_iv: "6fd9b707f27ec0545a23b0646a8c3ee7"
server_key_version: "encryptionkey@0"
I20260929 14:48:21.958240 10677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:21.974926 10677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:21.978454 10677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:21.978665 10677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:21.979287 10677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:21.980195 10677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:21.980263 10677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:21.980330 10677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:21.980374 10677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:21.990944 10677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40995
I20260929 14:48:21.990968 10805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40995 every 8 connection(s)
I20260929 14:48:21.991942 10677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:48:21.998685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10677
I20260929 14:48:21.998807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:48:21.999218 392 external_mini_cluster.cc:1428] Setting key 285d1221726916200192dde7d23350d4
Sep 29 14:48:21 dist-test-slave-5hd1 krb5kdc[10570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:22.009994 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:21.994376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38139 (local address 127.0.98.62:42115)
0929 14:48:21.994541 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:21.994544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:21.995761 (+ 1217us) server_negotiation.cc:407] Connection header received
0929 14:48:21.996805 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:21.996809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:21.996860 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:21.996951 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:21.998146 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:21.998831 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.000668 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.001063 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.001176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.005028 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:22.005035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:22.005041 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:22.005077 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:22.007145 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.007836 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.007843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.007847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.007952 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.008437 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.008443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.008447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.008738 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:22.008878 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.009613 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.009764 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:48:22.011112 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:21.994776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.1:38139)
0929 14:48:21.995586 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:21.995633 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:21.996593 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:21.997161 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:21.997168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:21.997524 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:21.997999 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:21.998011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:21.999071 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:21.999076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.000503 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.000521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.001258 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.001263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.001518 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.002371 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:22.002400 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:22.004778 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:22.007309 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.007315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.007334 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.007683 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.008068 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.008070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.008071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.008308 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.008899 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:22.008906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:22.009268 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.009478 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.009851 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":163,"threads_started":1}
I20260929 14:48:22.012666 10806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260929 14:48:22.012997 10806 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:22.013679 10806 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:48:22.015882 10615 ts_manager.cc:194] Registered new tserver with Master: cd5de2fdb5d14550ad95369eb7f899f4 (127.0.98.1:40995)
I20260929 14:48:22.017189 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:48:22.032390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:22.163916 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:22.164296 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:22.164374 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:22.168880 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:22.169003 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:22.169106 10811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:48:22.175266 10811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:22.176895 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:22.178218 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:22.190387 10816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:22.190407 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:22.190477 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:22.190899 10811 server_base.cc:1034] running on GCE node
I20260929 14:48:22.191421 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:22.192395 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:22.193627 10811 hybrid_clock.cc:648] HybridClock initialized: now 1790693302193610 us; error 44 us; skew 500 ppm
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[10570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:22.203980 10811 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:48:22.205071 10811 webserver.cc:460] Webserver started at http://127.0.98.2:40955/ using document root <none> and password file <none>
I20260929 14:48:22.205663 10811 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:22.205729 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:22.205904 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:22.207984 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d84b6f1943e45178db6cc0099ec592b"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "aa764fcc35fe2d83e588303d6845f762"
server_key_iv: "182b7b3be6decf62f5e705579078d28a"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.208633 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d84b6f1943e45178db6cc0099ec592b"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "aa764fcc35fe2d83e588303d6845f762"
server_key_iv: "182b7b3be6decf62f5e705579078d28a"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.212983 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:48:22.215771 10826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.217314 10811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:48:22.217469 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "5d84b6f1943e45178db6cc0099ec592b"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "aa764fcc35fe2d83e588303d6845f762"
server_key_iv: "182b7b3be6decf62f5e705579078d28a"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.217589 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:22.237710 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:22.241180 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:22.241391 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:22.241992 10811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:22.242925 10811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:22.242990 10811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.243050 10811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:22.243070 10811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.253324 10811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37813
I20260929 14:48:22.253460 10939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37813 every 8 connection(s)
I20260929 14:48:22.254382 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:48:22.260741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10811
I20260929 14:48:22.260851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:48:22.261144 392 external_mini_cluster.cc:1428] Setting key 805c65e61fd407a9cfa21a17426fdd48
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[10570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:22.270802 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.256778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41613 (local address 127.0.98.62:42115)
0929 14:48:22.256980 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.256984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.258236 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 14:48:22.259190 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.259193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.259247 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.259327 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:22.260781 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.261526 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.263185 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.263445 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.263555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.266491 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:22.266494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:22.266496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:22.266523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:22.268237 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.268815 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.268820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.268825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.268947 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.269366 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.269372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.269375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.269685 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:22.269802 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.270365 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.270552 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 14:48:22.271526 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.257246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.2:41613)
0929 14:48:22.258048 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.258102 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.258980 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.259605 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.259613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.259979 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:22.260554 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.260575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.261730 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.261733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.263033 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.263059 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.263640 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.263647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.263889 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.264631 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:22.264658 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:22.266290 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:22.268392 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.268398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.268410 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.268701 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.269085 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.269088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.269089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.269254 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.269805 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:22.269811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:22.270083 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.270271 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.270552 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":159,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:48:22.272924 10940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260929 14:48:22.273288 10940 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:22.274019 10940 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260929 14:48:22.275504 10615 ts_manager.cc:194] Registered new tserver with Master: 5d84b6f1943e45178db6cc0099ec592b (127.0.98.2:37813)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:48:22.276333 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:48:22.293651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:48:22.410117 10945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:48:22.410393 10945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:48:22.410436 10945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:48:22.414690 10945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:48:22.414773 10945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:48:22.414881 10945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:48:22.420614 10945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:48:22.421989 10945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:48:22.423235 10945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:48:22.434145 10951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:48:22.434129 10953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:22.434201 10945 server_base.cc:1034] running on GCE node
W20260929 14:48:22.434129 10950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:48:22.435029 10945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:48:22.435859 10945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:48:22.437057 10945 hybrid_clock.cc:648] HybridClock initialized: now 1790693302437022 us; error 64 us; skew 500 ppm
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[10570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:48:22.440349 10945 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:48:22.441509 10945 webserver.cc:460] Webserver started at http://127.0.98.3:40065/ using document root <none> and password file <none>
I20260929 14:48:22.442193 10945 fs_manager.cc:359] Metadata directory not provided
I20260929 14:48:22.442286 10945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:48:22.442556 10945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:48:22.445181 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e689cb40869409bb8c35ad3193cf344"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "da836e2cf1e8bca5a3ac7e6f67b86f37"
server_key_iv: "333e341586caec4521ed3c885097e97d"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.445844 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e689cb40869409bb8c35ad3193cf344"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "da836e2cf1e8bca5a3ac7e6f67b86f37"
server_key_iv: "333e341586caec4521ed3c885097e97d"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.450700 10945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:48:22.453715 10960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.455274 10945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:48:22.455533 10945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "7e689cb40869409bb8c35ad3193cf344"
format_stamp: "Formatted at 2026-09-29 14:48:22 on dist-test-slave-5hd1"
server_key: "da836e2cf1e8bca5a3ac7e6f67b86f37"
server_key_iv: "333e341586caec4521ed3c885097e97d"
server_key_version: "encryptionkey@0"
I20260929 14:48:22.455667 10945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:48:22.478338 10945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:48:22.482286 10945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:48:22.482525 10945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:48:22.483273 10945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:48:22.484364 10945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:48:22.484432 10945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.484505 10945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:48:22.484551 10945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:48:22.496488 10945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39785
I20260929 14:48:22.496517 11073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39785 every 8 connection(s)
I20260929 14:48:22.497645 10945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:48:22.501742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 10945
I20260929 14:48:22.501891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:48:22.502274 392 external_mini_cluster.cc:1428] Setting key f0a94406dbc2968f898654454d92451d
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[10570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:48:22.513001 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.499873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57553 (local address 127.0.98.62:42115)
0929 14:48:22.500102 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.500108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.501566 (+ 1458us) server_negotiation.cc:407] Connection header received
0929 14:48:22.502824 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.502827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.502912 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.503014 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:22.504691 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.505284 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.506508 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.506731 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.506812 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.509342 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:22.509346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:22.509348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:22.509373 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:22.510810 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.511300 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.511303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.511304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.511383 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.511740 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.511744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.511745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.511978 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:22.512096 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.512668 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.512820 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:48:22.513756 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.500481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42115 (local address 127.0.98.3:57553)
0929 14:48:22.501393 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.501445 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.502552 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.503241 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.503251 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.503797 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:48:22.504533 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.504553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.505446 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.505450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.506388 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.506398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.506872 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.506874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.507012 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.507646 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:48:22.507663 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:48:22.509182 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:48:22.510919 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.510924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.510935 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.511200 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.511479 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:48:22.511480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:48:22.511482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:48:22.511605 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:48:22.512085 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:48:22.512092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:48:22.512408 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.512595 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.512919 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":150,"threads_started":1}
I20260929 14:48:22.515209 11074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42115
I20260929 14:48:22.515480 11074 heartbeater.cc:464] Registering TS with master...
I20260929 14:48:22.515992 11074 heartbeater.cc:511] Master 127.0.98.62:42115 requested a full tablet report, sending...
I20260929 14:48:22.517172 10615 ts_manager.cc:194] Registered new tserver with Master: 7e689cb40869409bb8c35ad3193cf344 (127.0.98.3:39785)
I20260929 14:48:22.517812 10615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57553
I20260929 14:48:22.528996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:48:22.539215 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.531283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.0.98.62:42115)
0929 14:48:22.531462 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.531466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.531593 (+ 127us) server_negotiation.cc:407] Connection header received
0929 14:48:22.531677 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.531679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.531734 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.531819 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:48:22.532743 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.533482 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.534382 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.534636 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.534728 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.536074 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:48:22.536077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:48:22.536079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:48:22.536098 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:48:22.537636 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.538040 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.538042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.538044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.538119 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:48:22.538348 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:48:22.538351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:48:22.538352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:48:22.538591 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:48:22.538701 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.538959 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.539068 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 14:48:22.542450 10615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:48:22.545776 10615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:48:22.565307 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.558390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37813 (local address 127.0.0.1:58442)
0929 14:48:22.558905 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.558982 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.559158 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.560132 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.560139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.560193 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:48:22.560354 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.560365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.561879 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.561891 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.563348 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.563603 (+ 255us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.564701 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.564704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.564821 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.564859 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.564987 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.565134 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":122,"threads_started":1}
I20260929 14:48:22.565321 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.558600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.0.98.2:37813)
0929 14:48:22.559134 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.559149 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.559180 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:48:22.559414 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.559421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.559699 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.559895 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:48:22.560569 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.561752 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.563726 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.564575 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.564697 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.564877 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.565020 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.565126 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":128,"threads_started":1}
I20260929 14:48:22.565562 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.559502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39785 (local address 127.0.0.1:60792)
0929 14:48:22.560384 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.560396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.560473 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.560951 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.560954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.560972 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:48:22.561141 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.561152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.562749 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.562752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.563894 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.563903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.565064 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.565068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.565192 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.565226 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.565282 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.565385 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":119,"threads_started":1}
I20260929 14:48:22.565891 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.559378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.98.1:40995)
0929 14:48:22.560762 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.560768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.560789 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:48:22.560854 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.560859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.561076 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.561246 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:48:22.561484 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.562613 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.564104 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.564918 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.565039 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.565202 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.565646 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.565760 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":160,"threads_started":1}
I20260929 14:48:22.565567 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.559177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40995 (local address 127.0.0.1:51110)
0929 14:48:22.559441 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.559456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.559555 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.561213 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.561216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.561234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:48:22.561384 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.561393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.562776 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.562785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.563968 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.563981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.565259 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.565265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.565347 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.565363 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.565424 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.565482 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260929 14:48:22.566229 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.559965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.98.3:39785)
0929 14:48:22.560348 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.560443 (+ 95us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.560472 (+ 29us) server_negotiation.cc:407] Connection header received
0929 14:48:22.560612 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.560618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.560793 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.560973 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:48:22.561338 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.562613 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.564020 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.564918 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.565775 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.565905 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.565959 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.566067 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260929 14:48:22.569571 10740 tablet_service.cc:1467] Processing CreateTablet for tablet d590032a8f084057b97bdcbd7014d189 (DEFAULT_TABLE table=test-table [id=e882caffeb6e4900ae7fb168d9513625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:48:22.569679 11008 tablet_service.cc:1467] Processing CreateTablet for tablet d590032a8f084057b97bdcbd7014d189 (DEFAULT_TABLE table=test-table [id=e882caffeb6e4900ae7fb168d9513625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:48:22.569679 10874 tablet_service.cc:1467] Processing CreateTablet for tablet d590032a8f084057b97bdcbd7014d189 (DEFAULT_TABLE table=test-table [id=e882caffeb6e4900ae7fb168d9513625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:48:22.570937 10740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590032a8f084057b97bdcbd7014d189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:22.571062 11008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590032a8f084057b97bdcbd7014d189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:22.571062 10874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d590032a8f084057b97bdcbd7014d189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:48:22.579423 11091 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Bootstrap starting.
I20260929 14:48:22.580340 11092 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Bootstrap starting.
I20260929 14:48:22.580580 11093 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Bootstrap starting.
I20260929 14:48:22.582319 11091 tablet_bootstrap.cc:654] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:22.582319 11092 tablet_bootstrap.cc:654] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:22.582962 11093 tablet_bootstrap.cc:654] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Neither blocks nor log segments found. Creating new log.
I20260929 14:48:22.583428 11091 log.cc:824] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:22.583580 11092 log.cc:824] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:22.583772 11093 log.cc:824] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:48:22.585711 11092 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: No bootstrap required, opened a new log
I20260929 14:48:22.585815 11093 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: No bootstrap required, opened a new log
I20260929 14:48:22.585830 11091 tablet_bootstrap.cc:492] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: No bootstrap required, opened a new log
I20260929 14:48:22.585975 11092 ts_tablet_manager.cc:1397] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:48:22.586023 11093 ts_tablet_manager.cc:1397] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:48:22.586051 11091 ts_tablet_manager.cc:1397] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:48:22.589802 11092 raft_consensus.cc:348] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.590122 11092 raft_consensus.cc:374] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:22.590204 11092 raft_consensus.cc:729] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e689cb40869409bb8c35ad3193cf344, State: Initialized, Role: FOLLOWER
I20260929 14:48:22.590101 11093 raft_consensus.cc:348] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.590400 11093 raft_consensus.cc:374] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:22.590513 11093 raft_consensus.cc:729] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d84b6f1943e45178db6cc0099ec592b, State: Initialized, Role: FOLLOWER
I20260929 14:48:22.590626 11091 raft_consensus.cc:348] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.590865 11091 raft_consensus.cc:374] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:48:22.590794 11092 consensus_queue.cc:260] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.590972 11091 raft_consensus.cc:729] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5de2fdb5d14550ad95369eb7f899f4, State: Initialized, Role: FOLLOWER
I20260929 14:48:22.591078 11093 consensus_queue.cc:260] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.591375 11091 consensus_queue.cc:260] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.591642 11074 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260929 14:48:22.592077 11092 ts_tablet_manager.cc:1428] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:48:22.592135 11093 ts_tablet_manager.cc:1428] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:48:22.592211 11091 ts_tablet_manager.cc:1428] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:48:22.592519 10940 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
I20260929 14:48:22.592811 10806 heartbeater.cc:503] Master 127.0.98.62:42115 was elected leader, sending a full tablet report...
W20260929 14:48:22.743585 10807 tablet.cc:2369] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:48:22.748732 11075 tablet.cc:2369] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:48:22.755703 10941 tablet.cc:2369] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:48:22.789551 11097 raft_consensus.cc:482] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:48:22.789794 11097 raft_consensus.cc:504] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.791235 11097 leader_election.cc:288] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d84b6f1943e45178db6cc0099ec592b (127.0.98.2:37813), cd5de2fdb5d14550ad95369eb7f899f4 (127.0.98.1:40995)
I20260929 14:48:22.796785 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.791526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37813 (local address 127.0.98.3:58335)
0929 14:48:22.791744 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.791766 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.791940 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.792376 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.792380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.792394 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:48:22.792538 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.792543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.793646 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.793649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.795207 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.795214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.796121 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.796126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.796301 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.796327 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.796476 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.796589 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":86}
I20260929 14:48:22.796764 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.791661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58335 (local address 127.0.98.2:37813)
0929 14:48:22.791830 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.791837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.791856 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:48:22.792108 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.792112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.792230 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.792344 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:48:22.792665 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.793507 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.795405 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.795963 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.796099 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.796164 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.796493 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.796596 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:48:22.796854 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.791862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40995 (local address 127.0.98.3:40403)
0929 14:48:22.792266 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.792278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:48:22.792376 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:48:22.792677 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:48:22.792680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:48:22.792696 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:48:22.792779 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.792783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.793753 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.793756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.795204 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:48:22.795214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.796302 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.796305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:48:22.796408 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.796432 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:48:22.796552 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 14:48:22.796645 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":50,"thread_start_us":168,"threads_started":1}
I20260929 14:48:22.796835 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.792064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40403 (local address 127.0.98.1:40995)
0929 14:48:22.792322 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.792328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.792348 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:48:22.792501 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.792504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.792557 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.792643 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:48:22.792883 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.793633 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.795354 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.796002 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.796129 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.796244 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.796591 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.796718 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:48:22.797634 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590032a8f084057b97bdcbd7014d189" candidate_uuid: "7e689cb40869409bb8c35ad3193cf344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" is_pre_election: true
I20260929 14:48:22.797704 10894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590032a8f084057b97bdcbd7014d189" candidate_uuid: "7e689cb40869409bb8c35ad3193cf344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d84b6f1943e45178db6cc0099ec592b" is_pre_election: true
I20260929 14:48:22.797969 10760 raft_consensus.cc:2454] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e689cb40869409bb8c35ad3193cf344 in term 0.
I20260929 14:48:22.798079 10894 raft_consensus.cc:2454] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e689cb40869409bb8c35ad3193cf344 in term 0.
I20260929 14:48:22.798553 10964 leader_election.cc:302] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e689cb40869409bb8c35ad3193cf344, cd5de2fdb5d14550ad95369eb7f899f4; no voters:
I20260929 14:48:22.798866 11097 raft_consensus.cc:2781] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:48:22.798995 11097 raft_consensus.cc:482] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:48:22.799062 11097 raft_consensus.cc:3037] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:22.800487 11097 raft_consensus.cc:504] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.800984 11097 leader_election.cc:288] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [CANDIDATE]: Term 1 election: Requested vote from peers 5d84b6f1943e45178db6cc0099ec592b (127.0.98.2:37813), cd5de2fdb5d14550ad95369eb7f899f4 (127.0.98.1:40995)
I20260929 14:48:22.801435 10894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590032a8f084057b97bdcbd7014d189" candidate_uuid: "7e689cb40869409bb8c35ad3193cf344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d84b6f1943e45178db6cc0099ec592b"
I20260929 14:48:22.801505 10760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d590032a8f084057b97bdcbd7014d189" candidate_uuid: "7e689cb40869409bb8c35ad3193cf344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5de2fdb5d14550ad95369eb7f899f4"
I20260929 14:48:22.801590 10894 raft_consensus.cc:3037] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:22.801672 10760 raft_consensus.cc:3037] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:48:22.802917 10894 raft_consensus.cc:2454] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e689cb40869409bb8c35ad3193cf344 in term 1.
I20260929 14:48:22.802955 10760 raft_consensus.cc:2454] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e689cb40869409bb8c35ad3193cf344 in term 1.
I20260929 14:48:22.803220 10964 leader_election.cc:302] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d84b6f1943e45178db6cc0099ec592b, 7e689cb40869409bb8c35ad3193cf344; no voters:
I20260929 14:48:22.803400 11097 raft_consensus.cc:2781] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:48:22.803716 11097 raft_consensus.cc:686] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 1 LEADER]: Becoming Leader. State: Replica: 7e689cb40869409bb8c35ad3193cf344, State: Running, Role: LEADER
I20260929 14:48:22.804150 11097 consensus_queue.cc:237] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } }
I20260929 14:48:22.807916 10615 catalog_manager.cc:5495] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e689cb40869409bb8c35ad3193cf344 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7e689cb40869409bb8c35ad3193cf344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e689cb40869409bb8c35ad3193cf344" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39785 } health_report { overall_health: HEALTHY } } }
I20260929 14:48:22.862164 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.858176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.0.98.3:39785)
0929 14:48:22.858318 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:48:22.858322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:48:22.858339 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:48:22.858436 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:48:22.858439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:48:22.858503 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:48:22.858630 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:48:22.858868 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.859425 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.860412 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:48:22.860636 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:48:22.860723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:48:22.861082 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:48:22.861664 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:48:22.861762 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:48:22.861931 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 14:48:22.861990 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:48:22.869161 10760 raft_consensus.cc:1261] T d590032a8f084057b97bdcbd7014d189 P cd5de2fdb5d14550ad95369eb7f899f4 [term 1 FOLLOWER]: Refusing update from remote peer 7e689cb40869409bb8c35ad3193cf344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:48:22.869266 10894 raft_consensus.cc:1261] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 1 FOLLOWER]: Refusing update from remote peer 7e689cb40869409bb8c35ad3193cf344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:48:22.870040 11101 consensus_queue.cc:1035] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5de2fdb5d14550ad95369eb7f899f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:48:22.870280 11097 consensus_queue.cc:1035] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d84b6f1943e45178db6cc0099ec592b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:48:22.878648 11106 mvcc.cc:205] Tried to move back new op lower bound from 7334679768542867456 to 7334679768289124352. Current Snapshot: MvccSnapshot[applied={T|T < 7334679768542867456}]
I20260929 14:48:22.879094 11105 mvcc.cc:205] Tried to move back new op lower bound from 7334679768542867456 to 7334679768289124352. Current Snapshot: MvccSnapshot[applied={T|T < 7334679768542867456}]
I20260929 14:48:22.886809 10615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:48:22.887032 10615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:48:22.889744 10615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c77d1e2229334c44b14a796f02e6eecc: Sending DeleteTablet for 3 replicas of tablet d590032a8f084057b97bdcbd7014d189
I20260929 14:48:22.890544 10874 tablet_service.cc:1514] Processing DeleteTablet for tablet d590032a8f084057b97bdcbd7014d189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:48:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58442
I20260929 14:48:22.890653 11008 tablet_service.cc:1514] Processing DeleteTablet for tablet d590032a8f084057b97bdcbd7014d189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:48:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60792
I20260929 14:48:22.891173 10740 tablet_service.cc:1514] Processing DeleteTablet for tablet d590032a8f084057b97bdcbd7014d189 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:48:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51110
I20260929 14:48:22.891245 11114 tablet_replica.cc:330] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: stopping tablet replica
I20260929 14:48:22.891419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10677
I20260929 14:48:22.893144 11114 raft_consensus.cc:2229] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:48:22.893579 11114 raft_consensus.cc:2258] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:48:22.894981 11115 tablet_replica.cc:330] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: stopping tablet replica
I20260929 14:48:22.895269 11114 ts_tablet_manager.cc:1905] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:48:22.895303 11115 raft_consensus.cc:2229] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:48:22.895738 11115 raft_consensus.cc:2258] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:48:22.897562 11115 ts_tablet_manager.cc:1905] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:48:22.898479 11114 ts_tablet_manager.cc:1918] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:48:22.898785 11114 log.cc:1196] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/d590032a8f084057b97bdcbd7014d189
I20260929 14:48:22.899278 11114 ts_tablet_manager.cc:1939] T d590032a8f084057b97bdcbd7014d189 P 5d84b6f1943e45178db6cc0099ec592b: Deleting consensus metadata
I20260929 14:48:22.901186 10602 catalog_manager.cc:4841] TS 5d84b6f1943e45178db6cc0099ec592b (127.0.98.2:37813): tablet d590032a8f084057b97bdcbd7014d189 (table test-table [id=e882caffeb6e4900ae7fb168d9513625]) successfully deleted
I20260929 14:48:22.901496 11115 ts_tablet_manager.cc:1918] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:48:22.901649 11115 log.cc:1196] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/d590032a8f084057b97bdcbd7014d189
W20260929 14:48:22.901724 10602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40995 (error 108)
I20260929 14:48:22.902031 11115 ts_tablet_manager.cc:1939] T d590032a8f084057b97bdcbd7014d189 P 7e689cb40869409bb8c35ad3193cf344: Deleting consensus metadata
I20260929 14:48:22.902714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10811
I20260929 14:48:22.903707 10602 catalog_manager.cc:4841] TS 7e689cb40869409bb8c35ad3193cf344 (127.0.98.3:39785): tablet d590032a8f084057b97bdcbd7014d189 (table test-table [id=e882caffeb6e4900ae7fb168d9513625]) successfully deleted
I20260929 14:48:22.904570 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:48:22.903075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40995 (local address 127.0.0.1:51126)
0929 14:48:22.904368 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0929 14:48:22.904452 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1167}
W20260929 14:48:22.905004 10602 catalog_manager.cc:4584] TS cd5de2fdb5d14550ad95369eb7f899f4 (127.0.98.1:40995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d590032a8f084057b97bdcbd7014d189: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40995: connect: Connection refused (error 111)
I20260929 14:48:22.910866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10945
I20260929 14:48:22.918323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 10583
2026-09-29T14:48:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2660 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[11120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[11120](info): set up 2 sockets
Sep 29 14:48:22 dist-test-slave-5hd1 krb5kdc[11120](info): commencing operation
krb5kdc: starting...
W20260929 14:48:23.977527 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:48:24 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:48:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:48:24.171934 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiGqqz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:48:27.001 UTC [11150] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:48:27.001 UTC [11150] LOG: listening on IPv4 address "127.0.98.20", port 52831
2026-09-29 14:48:27.005 UTC [11150] LOG: listening on Unix socket "/tmp/.s.PGSQL.52831"
2026-09-29 14:48:27.028 UTC [11153] LOG: database system was shut down at 2026-09-29 14:48:25 UTC
2026-09-29 14:48:27.032 UTC [11150] LOG: database system is ready to accept connections
I20260929 14:48:27.993100 392 mini_postgres.cc:96] Postgres bound to 52831
2026-09-29 14:48:28.000 UTC [11161] FATAL: database "slave" does not exist
127.0.98.20:52831 - accepting connections
I20260929 14:48:28.001019 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:48:28.017367 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:48:28.567692 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:48:28.567859 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin
2026-09-29 14:48:28,874 [I] DB FLAVOR :POSTGRES
2026-09-29 14:48:28,874 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:48:28,874 [I] Checking connection..
2026-09-29 14:48:28,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:48:29,142 [I] Checking connection passed.
2026-09-29 14:48:29,142 [I] --------- Verifying version history table ---------
2026-09-29 14:48:29,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:48:29,415 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:48:29,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:48:29,707 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:48:29,707 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:48:29,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:48:29.952 UTC [11225] WARNING: there is no transaction in progress
2026-09-29 14:48:29,959 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:48:29,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:48:30,220 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:48:30,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:48:30,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:48:30,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:48:30,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5hd1.c.gcp-upstream.internal','N') ;"
2026-09-29 14:48:30,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:48:31,244 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:48:31,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:48:31,508 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:48:31,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:48:31,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:48:32,027 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:48:32,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:48:33.187 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.206 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.228 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.246 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.264 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.284 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.348 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.356 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.373 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.385 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.396 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.413 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.424 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.431 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.442 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:33.451 UTC [11333] WARNING: there is no transaction in progress
2026-09-29 14:48:34,270 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:48:34,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:48:34,565 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:48:34,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:48:34,868 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:48:34,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:48:35,157 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:48:35,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:48:35,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:48:35,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5hd1.c.gcp-upstream.internal';"
2026-09-29 14:48:35,981 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:48:35,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:48:36,270 [I] DB_PATCHES have already been applied
I20260929 14:48:36.276337 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiGqqz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:48:36.276448 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:48:36.280613 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55441
Sep 29, 2026 2:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:48:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:48:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:48:37.415550 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.415s user 0.004s sys 0.009s
Sep 29, 2026 2:48:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:48:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 2:48:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:48:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:49:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:49:00.725055 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:49:00 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693340, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:49:00.818859 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:49:00.829335 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:49:01.863983 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:49:01.864101 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms
I20260929 14:49:01.864460 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52831
2026-09-29 14:49:02,092 [I] DB FLAVOR :POSTGRES
2026-09-29 14:49:02,092 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:49:02,093 [I] Checking connection
2026-09-29 14:49:02,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:49:02,366 [I] connection success
2026-09-29 14:49:02,366 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:49:02,366 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:49:02,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:49:02,664 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:49:02,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:49:02,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:49:02,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52831/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:49:02,992 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:49:03.561818 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:49:03.794358 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:49:03.851922 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:49:03 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693340, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:04.296270 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:49:04.296445 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:49:04.296474 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:49:04.300207 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11514
I20260929 14:49:04.300355 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33091
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:49:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:49:06.566182 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.747s user 0.008s sys 0.203s
I20260929 14:49:06.566449 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:49:06.566892 392 mini_ranger_kms.cc:401] 127.0.98.20:33091/kms/v1/keys
Sep 29 14:49:06 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:49:07 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:49:07 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693346, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:49:10 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:49:10.057197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39423
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39423
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33091/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:49:10.171382 11564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:49:10.171828 11564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:49:10.171898 11564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:49:10.176313 11564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:49:10.176401 11564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:49:10.176437 11564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:49:10.176479 11564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:49:10.176514 11564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:49:10.182044 11564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33091/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39423
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:49:10.183166 11564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:49:10.184211 11564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:49:10.193368 11569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:10.193357 11572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:10.193492 11570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:49:10.193807 11564 server_base.cc:1034] running on GCE node
I20260929 14:49:10.194480 11564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:49:10.195603 11564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:49:10.196871 11564 hybrid_clock.cc:648] HybridClock initialized: now 1790693350196836 us; error 60 us; skew 500 ppm
Sep 29 14:49:10 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:49:10.199666 11564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:49:10.200892 11564 webserver.cc:460] Webserver started at http://127.0.98.62:46061/ using document root <none> and password file <none>
I20260929 14:49:10.201539 11564 fs_manager.cc:359] Metadata directory not provided
I20260929 14:49:10.201599 11564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:49:10.201838 11564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:49:10 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:10.260062 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "dff27961bb4f4d22bc0cf32253ebbcb1"
format_stamp: "Formatted at 2026-09-29 14:49:10 on dist-test-slave-5hd1"
server_key: "b9183f26b6ee4ba20ea0b1c466b5a9f6"
server_key_iv: "f302512593720e44b0afd26ffc392415"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:10.261165 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "dff27961bb4f4d22bc0cf32253ebbcb1"
format_stamp: "Formatted at 2026-09-29 14:49:10 on dist-test-slave-5hd1"
server_key: "b9183f26b6ee4ba20ea0b1c466b5a9f6"
server_key_iv: "f302512593720e44b0afd26ffc392415"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:10.267611 11564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 14:49:10.286099 11579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:10.288259 11564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:49:10.288475 11564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "dff27961bb4f4d22bc0cf32253ebbcb1"
format_stamp: "Formatted at 2026-09-29 14:49:10 on dist-test-slave-5hd1"
server_key: "b9183f26b6ee4ba20ea0b1c466b5a9f6"
server_key_iv: "f302512593720e44b0afd26ffc392415"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:10.288682 11564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:49:10.311806 11564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:49:10.316285 11564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:49:10.316560 11564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:49:10.328260 11564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39423
I20260929 14:49:10.328323 11631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39423 every 8 connection(s)
I20260929 14:49:10.329725 11564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:49:10.333026 11632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:49:10.337764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 11564
I20260929 14:49:10.337911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:49:10.338212 392 external_mini_cluster.cc:1428] Setting key 9332150c9cc46188248a9bee4c9f83dc
I20260929 14:49:10.339548 11632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: Bootstrap starting.
I20260929 14:49:10.341924 11632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: Neither blocks nor log segments found. Creating new log.
I20260929 14:49:10.342839 11632 log.cc:824] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:49:10.345460 11632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: No bootstrap required, opened a new log
I20260929 14:49:10.349583 11632 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260929 14:49:10.349858 11632 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:49:10.349959 11632 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff27961bb4f4d22bc0cf32253ebbcb1, State: Initialized, Role: FOLLOWER
Sep 29 14:49:10 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:49:10.350471 11632 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260929 14:49:10.350661 11632 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:49:10.350739 11632 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:49:10.350868 11632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:49:10.351881 11632 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260929 14:49:10.352324 11632 leader_election.cc:302] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff27961bb4f4d22bc0cf32253ebbcb1; no voters:
I20260929 14:49:10.352691 11632 leader_election.cc:288] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:49:10.352770 11637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:49:10.352981 11637 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [term 1 LEADER]: Becoming Leader. State: Replica: dff27961bb4f4d22bc0cf32253ebbcb1, State: Running, Role: LEADER
I20260929 14:49:10.353277 11637 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260929 14:49:10.353981 11632 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:49:10.354449 11637 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff27961bb4f4d22bc0cf32253ebbcb1. Latest consensus state: current_term: 1 leader_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } } }
I20260929 14:49:10.354578 11637 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:49:10.354796 11637 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff27961bb4f4d22bc0cf32253ebbcb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } } }
I20260929 14:49:10.355014 11637 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:49:10.356299 11632 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:49:10.356600 11641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:49:10.360476 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:10.339792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.0.98.62:39423)
0929 14:49:10.340443 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:10.340471 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:10.340542 (+ 71us) server_negotiation.cc:407] Connection header received
0929 14:49:10.341651 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:10.341684 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:10.342271 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:10.342879 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:49:10.343903 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:10.345377 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:10.346844 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:10.347556 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:10.347723 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:10.351294 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:49:10.351303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:49:10.351328 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:49:10.351371 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:49:10.353956 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:10.354685 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:10.354692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:10.354699 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:10.354857 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:10.355177 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:10.355180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:10.355182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:10.355830 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:49:10.356027 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:10.357068 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0929 14:49:10.357417 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":157,"threads_started":1}
I20260929 14:49:10.364841 11641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:49:10.379547 11641 catalog_manager.cc:1347] Generated new cluster ID: 3a7f2329bc004fe58519c1c8fbd74d98
I20260929 14:49:10.379627 11641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:49:10.434648 11641 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:49:10.436082 11641 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:49:10.443396 11641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: Generated new TSK 0
I20260929 14:49:10.444264 11641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:49:11.252130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:49:11.391667 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:49:11.391935 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:49:11.391976 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:49:11.396452 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:49:11.396531 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:49:11.396631 11677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:49:11.401935 11677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:49:11.403002 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:49:11.403966 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:49:11.413992 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:11.414042 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:11.413992 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:49:11.414505 11677 server_base.cc:1034] running on GCE node
I20260929 14:49:11.414986 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:49:11.415598 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:49:11.416935 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790693351416909 us; error 51 us; skew 500 ppm
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:49:11.419790 11677 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:49:11.420783 11677 webserver.cc:460] Webserver started at http://127.0.98.1:43939/ using document root <none> and password file <none>
I20260929 14:49:11.421463 11677 fs_manager.cc:359] Metadata directory not provided
I20260929 14:49:11.421553 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:49:11.421757 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:11.442705 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "690f1578bf93499eb1e713f25e5c10c2"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "885ecf055c5832207be0aac6fd36c57c"
server_key_iv: "add0a98fd165d85ea28efbeefda15611"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.443663 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "690f1578bf93499eb1e713f25e5c10c2"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "885ecf055c5832207be0aac6fd36c57c"
server_key_iv: "add0a98fd165d85ea28efbeefda15611"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.453076 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20260929 14:49:11.478752 11694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:11.480674 11677 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 14:49:11.480944 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "690f1578bf93499eb1e713f25e5c10c2"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "885ecf055c5832207be0aac6fd36c57c"
server_key_iv: "add0a98fd165d85ea28efbeefda15611"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.481235 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:49:11.499197 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:49:11.503413 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:49:11.503688 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:49:11.504588 11677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:49:11.505915 11677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:49:11.506057 11677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 14:49:11.506224 11677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:49:11.506328 11677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:11.524893 11677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35039
I20260929 14:49:11.526544 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:49:11.528116 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35039 every 8 connection(s)
I20260929 14:49:11.536767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 11677
I20260929 14:49:11.536929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:49:11.537346 392 external_mini_cluster.cc:1428] Setting key a274e52f7672180a51ca80ecd71cef56
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:49:11.550159 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:11.530884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39423 (local address 127.0.98.1:55955)
0929 14:49:11.531485 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:11.531547 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:11.533057 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:11.534065 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:11.534076 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:11.534494 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:49:11.535302 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:11.535323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.536753 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.536760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:11.538637 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:11.538649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.539427 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.539431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:11.539653 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:11.540530 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:49:11.540554 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:49:11.542544 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:49:11.545573 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:11.545580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:11.545609 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:11.546063 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:11.546718 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:11.546723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:11.546725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:11.547118 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:11.547793 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:49:11.547803 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:49:11.548300 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 14:49:11.548556 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 14:49:11.549011 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":182,"threads_started":1}
I20260929 14:49:11.551204 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:11.532261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55955 (local address 127.0.98.62:39423)
0929 14:49:11.533539 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:11.533545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:11.533560 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:49:11.533626 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:11.533630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:11.533698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:11.535465 (+ 1767us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:49:11.535654 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.536576 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.538787 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.539135 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.539230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:11.542888 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:49:11.542895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:49:11.542903 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:49:11.542963 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:49:11.545335 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:11.546412 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:11.546418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:11.546420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:11.546542 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:11.547286 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:11.547290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:11.547292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:11.547591 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:49:11.547709 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:11.550821 (+ 3112us) server_negotiation.cc:299] Negotiation successful
0929 14:49:11.550991 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:49:11.552752 11808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39423
I20260929 14:49:11.553164 11808 heartbeater.cc:464] Registering TS with master...
I20260929 14:49:11.553951 11808 heartbeater.cc:511] Master 127.0.98.62:39423 requested a full tablet report, sending...
I20260929 14:49:11.556222 11596 ts_manager.cc:194] Registered new tserver with Master: 690f1578bf93499eb1e713f25e5c10c2 (127.0.98.1:35039)
I20260929 14:49:11.557833 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:49:11.580860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:49:11.731596 11816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:49:11.731947 11816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:49:11.732025 11816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:49:11.736613 11816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:49:11.736712 11816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:49:11.736824 11816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:49:11.743824 11816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:49:11.745528 11816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:49:11.746827 11816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:49:11.759793 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:11.759902 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:11.760456 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:49:11.760783 11816 server_base.cc:1034] running on GCE node
I20260929 14:49:11.761493 11816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:49:11.762449 11816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:49:11.763758 11816 hybrid_clock.cc:648] HybridClock initialized: now 1790693351763734 us; error 44 us; skew 500 ppm
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:49:11.773234 11816 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:49:11.774659 11816 webserver.cc:460] Webserver started at http://127.0.98.2:35877/ using document root <none> and password file <none>
I20260929 14:49:11.775504 11816 fs_manager.cc:359] Metadata directory not provided
I20260929 14:49:11.775640 11816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:49:11.775923 11816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:11.806059 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "f714318c184243afb4cdd3aeecccf1c1"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "14158e3a837d1743ecb8a5b8aeb8a80b"
server_key_iv: "5cfa7067e8fa6eb6582510f2f4bb464e"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.807046 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f714318c184243afb4cdd3aeecccf1c1"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "14158e3a837d1743ecb8a5b8aeb8a80b"
server_key_iv: "5cfa7067e8fa6eb6582510f2f4bb464e"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.813313 11816 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 14:49:11.831784 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:11.833732 11816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:49:11.834190 11816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "f714318c184243afb4cdd3aeecccf1c1"
format_stamp: "Formatted at 2026-09-29 14:49:11 on dist-test-slave-5hd1"
server_key: "14158e3a837d1743ecb8a5b8aeb8a80b"
server_key_iv: "5cfa7067e8fa6eb6582510f2f4bb464e"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:11.834381 11816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:49:11.859460 11816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:49:11.863584 11816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:49:11.863823 11816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:49:11.865180 11816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:49:11.866473 11816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:49:11.866549 11816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:11.866611 11816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:49:11.866653 11816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:11.880414 11816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34259
I20260929 14:49:11.880604 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34259 every 8 connection(s)
I20260929 14:49:11.881767 11816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:49:11.891474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 11816
I20260929 14:49:11.891690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:49:11.892055 392 external_mini_cluster.cc:1428] Setting key 3e3fa410a9573d69c6928f9284928221
Sep 29 14:49:11 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:49:11.903432 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:11.884512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50107 (local address 127.0.98.62:39423)
0929 14:49:11.884697 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:11.884703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:11.886441 (+ 1738us) server_negotiation.cc:407] Connection header received
0929 14:49:11.887967 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:11.887971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:11.888041 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:11.888104 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:49:11.890085 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.890972 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.894394 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.894845 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.894950 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:11.898635 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:49:11.898639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:49:11.898644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:49:11.898684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:49:11.900692 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:11.901382 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:11.901386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:11.901389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:11.901478 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:11.901899 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:11.901903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:11.901905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:11.902146 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:49:11.902249 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:11.903068 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 14:49:11.903234 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:49:11.905236 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:11.885127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39423 (local address 127.0.98.2:50107)
0929 14:49:11.886218 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:11.886291 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:11.887664 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:11.888668 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:11.888679 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:11.889123 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:49:11.889900 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:11.889920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.892315 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.892334 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:11.894125 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:11.894145 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:11.895276 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:11.895280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:11.895521 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:11.896386 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:49:11.896409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:49:11.898409 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:49:11.900852 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:11.900858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:11.900876 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:11.901243 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:11.901591 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:11.901594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:11.901596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:11.901774 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:11.902337 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:49:11.902345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:49:11.902744 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 14:49:11.903638 (+ 894us) client_negotiation.cc:240] Negotiation successful
0929 14:49:11.904014 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"spinlock_wait_cycles":636160,"thread_start_us":193,"threads_started":1}
I20260929 14:49:11.906883 11946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39423
I20260929 14:49:11.907296 11946 heartbeater.cc:464] Registering TS with master...
I20260929 14:49:11.908079 11946 heartbeater.cc:511] Master 127.0.98.62:39423 requested a full tablet report, sending...
I20260929 14:49:11.909688 11596 ts_manager.cc:194] Registered new tserver with Master: f714318c184243afb4cdd3aeecccf1c1 (127.0.98.2:34259)
I20260929 14:49:11.910691 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:49:11.936141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:49:12.110977 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:49:12.111406 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:49:12.111470 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 29 14:49:12 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 14:49:12.117620 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:49:12.117728 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:49:12.117830 11951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:49:12.126047 11951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:49:12.127749 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:49:12.129305 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:49:12.144114 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:49:12.144704 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:49:12.146154 11951 server_base.cc:1034] running on GCE node
W20260929 14:49:12.146489 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:49:12.147119 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:49:12.148068 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:49:12.149471 11951 hybrid_clock.cc:648] HybridClock initialized: now 1790693352149365 us; error 90 us; skew 500 ppm
Sep 29 14:49:12 dist-test-slave-5hd1 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:49:12.153795 11951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:49:12.155153 11951 webserver.cc:460] Webserver started at http://127.0.98.3:45763/ using document root <none> and password file <none>
I20260929 14:49:12.155968 11951 fs_manager.cc:359] Metadata directory not provided
I20260929 14:49:12.156121 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:49:12.156467 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:49:12 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:12.176167 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "f297d44038424cb09d1a623d5563f221"
format_stamp: "Formatted at 2026-09-29 14:49:12 on dist-test-slave-5hd1"
server_key: "a338a5306b60c60577f187d5c34db040"
server_key_iv: "dc8223a17259499a7c4aee299a5018f1"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:12.177124 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f297d44038424cb09d1a623d5563f221"
format_stamp: "Formatted at 2026-09-29 14:49:12 on dist-test-slave-5hd1"
server_key: "a338a5306b60c60577f187d5c34db040"
server_key_iv: "dc8223a17259499a7c4aee299a5018f1"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:12.193411 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.000s
I20260929 14:49:12.216084 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:12.217960 11951 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260929 14:49:12.218102 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "f297d44038424cb09d1a623d5563f221"
format_stamp: "Formatted at 2026-09-29 14:49:12 on dist-test-slave-5hd1"
server_key: "a338a5306b60c60577f187d5c34db040"
server_key_iv: "dc8223a17259499a7c4aee299a5018f1"
server_key_version: "kuduclusterkey@0"
I20260929 14:49:12.218257 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:49:12.236295 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:49:12.240072 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:49:12.240357 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:49:12.241169 11951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:49:12.242344 11951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:49:12.242448 11951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:12.242579 11951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:49:12.242645 11951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:49:12.254225 11951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45039
I20260929 14:49:12.254429 12079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45039 every 8 connection(s)
I20260929 14:49:12.255549 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:49:12.256999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 11951
I20260929 14:49:12.257151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:49:12.257490 392 external_mini_cluster.cc:1428] Setting key 89128f1a414aec2f5ddbadffe9679a6a
Sep 29 14:49:12 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:49:12.276137 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.259398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44261 (local address 127.0.98.62:39423)
0929 14:49:12.259554 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.259559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.260865 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 14:49:12.262306 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.262311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.262381 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.262509 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:49:12.264116 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.265078 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.268707 (+ 3629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.268991 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.269179 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.272375 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:49:12.272378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:49:12.272380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:49:12.272404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:49:12.273870 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:12.274415 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:12.274429 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:12.274430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:12.274501 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:12.274901 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:12.274904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:12.274906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:12.275144 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:49:12.275275 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.275835 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.275963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:49:12.277104 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.259916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39423 (local address 127.0.98.3:44261)
0929 14:49:12.260692 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.260760 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.261994 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.262741 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.262751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.263162 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:49:12.263928 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.263946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.265233 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.265237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.268544 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.268564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.269243 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.269248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.269588 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.270352 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:49:12.270375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:49:12.272136 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:49:12.274006 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:12.274012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:12.274031 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:12.274306 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:12.274603 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:49:12.274606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:49:12.274608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:49:12.274787 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:49:12.275279 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:49:12.275288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:49:12.275582 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.275774 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.276105 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":200,"threads_started":1}
I20260929 14:49:12.278419 12080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39423
I20260929 14:49:12.278680 12080 heartbeater.cc:464] Registering TS with master...
I20260929 14:49:12.279194 12080 heartbeater.cc:511] Master 127.0.98.62:39423 requested a full tablet report, sending...
I20260929 14:49:12.280314 11596 ts_manager.cc:194] Registered new tserver with Master: f297d44038424cb09d1a623d5563f221 (127.0.98.3:45039)
I20260929 14:49:12.281002 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44261
I20260929 14:49:12.283519 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:49:12.293686 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.285890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.0.98.62:39423)
0929 14:49:12.286189 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.286194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.286456 (+ 262us) server_negotiation.cc:407] Connection header received
0929 14:49:12.286607 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.286610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.286656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.286741 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:49:12.287405 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.287933 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.288864 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.289088 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.289192 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.290534 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:49:12.290538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:49:12.290541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:49:12.290573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:49:12.291996 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:12.292483 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:12.292488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:12.292490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:12.292568 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:49:12.292853 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:49:12.292856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:49:12.292858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:49:12.293024 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:49:12.293159 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.293395 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.293504 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260929 14:49:12.296586 11596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:49:12.299671 11596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:49:12.317312 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.308791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35039 (local address 127.0.0.1:37732)
0929 14:49:12.310474 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.310489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.310605 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.312090 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.312093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.312109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.312219 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.312224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.313745 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.313751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.314931 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.314940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.316762 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.316766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.316891 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.316944 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.317024 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.317137 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"mutex_wait_us":122,"thread_start_us":77,"threads_started":1}
I20260929 14:49:12.317844 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.309575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.98.3:45039)
0929 14:49:12.310957 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.310966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.310985 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:49:12.311043 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.311047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.311252 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.311425 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.311712 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.313655 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.316058 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.316849 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.316939 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.317076 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.317595 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.317700 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":123,"threads_started":1}
I20260929 14:49:12.317512 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.308695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45039 (local address 127.0.0.1:56552)
0929 14:49:12.310170 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.310219 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.310778 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.311412 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.311418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.311438 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.311572 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.311578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.313768 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.313770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.315909 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.315917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.317130 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.317134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.317209 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.317228 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.317292 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.317379 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":829,"thread_start_us":109,"threads_started":1}
I20260929 14:49:12.320446 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.309507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.0.98.1:35039)
0929 14:49:12.311681 (+ 2174us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.311687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.311708 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:49:12.311780 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.311784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.311947 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.312357 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.312484 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.313613 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.315058 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.316629 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.319964 (+ 3335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.320118 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.320158 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.320319 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2067,"thread_start_us":91,"threads_started":1}
I20260929 14:49:12.322220 12014 tablet_service.cc:1467] Processing CreateTablet for tablet 234550db8974442f823ce870896503ad (DEFAULT_TABLE table=test-table [id=7772a6e9c37649d68a38cfa656aead8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:49:12.323673 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234550db8974442f823ce870896503ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:49:12.323381 11740 tablet_service.cc:1467] Processing CreateTablet for tablet 234550db8974442f823ce870896503ad (DEFAULT_TABLE table=test-table [id=7772a6e9c37649d68a38cfa656aead8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:49:12.324466 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.310578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34259 (local address 127.0.0.1:58284)
0929 14:49:12.317906 (+ 7328us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.317922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.317996 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.318510 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.318513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.318529 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.318597 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.318602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.321124 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.321127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.322381 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.322391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.323728 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.323734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.323866 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.323908 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.324100 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.324301 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7262,"thread_start_us":88,"threads_started":1}
I20260929 14:49:12.324625 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234550db8974442f823ce870896503ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:49:12.324978 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.310904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.2:34259)
0929 14:49:12.311368 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.314343 (+ 2975us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.318085 (+ 3742us) server_negotiation.cc:407] Connection header received
0929 14:49:12.318173 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.318180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.318362 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.318721 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.318863 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.319793 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.322569 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.323541 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.323650 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.323884 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.324693 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.324828 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":158,"threads_started":1}
I20260929 14:49:12.327589 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 234550db8974442f823ce870896503ad (DEFAULT_TABLE table=test-table [id=7772a6e9c37649d68a38cfa656aead8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:49:12.328596 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234550db8974442f823ce870896503ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:49:12.343231 12099 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Bootstrap starting.
I20260929 14:49:12.343662 12098 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: Bootstrap starting.
I20260929 14:49:12.344388 12100 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Bootstrap starting.
I20260929 14:49:12.346124 12098 tablet_bootstrap.cc:654] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: Neither blocks nor log segments found. Creating new log.
I20260929 14:49:12.346347 12100 tablet_bootstrap.cc:654] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Neither blocks nor log segments found. Creating new log.
I20260929 14:49:12.347049 12098 log.cc:824] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:49:12.348001 12100 log.cc:824] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:49:12.349879 12099 tablet_bootstrap.cc:654] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Neither blocks nor log segments found. Creating new log.
I20260929 14:49:12.350504 12100 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: No bootstrap required, opened a new log
I20260929 14:49:12.350703 12100 ts_tablet_manager.cc:1397] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:49:12.350821 12099 log.cc:824] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Log is configured to *not* fsync() on all Append() calls
I20260929 14:49:12.352758 12099 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: No bootstrap required, opened a new log
I20260929 14:49:12.352916 12099 ts_tablet_manager.cc:1397] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 14:49:12.355854 12100 raft_consensus.cc:348] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.356112 12100 raft_consensus.cc:374] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:49:12.356189 12100 raft_consensus.cc:729] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f714318c184243afb4cdd3aeecccf1c1, State: Initialized, Role: FOLLOWER
I20260929 14:49:12.356650 12100 consensus_queue.cc:260] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.357494 12099 raft_consensus.cc:348] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.357707 12099 raft_consensus.cc:374] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:49:12.357753 12099 raft_consensus.cc:729] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f297d44038424cb09d1a623d5563f221, State: Initialized, Role: FOLLOWER
I20260929 14:49:12.358161 12100 ts_tablet_manager.cc:1428] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:49:12.358443 12099 consensus_queue.cc:260] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.358670 11946 heartbeater.cc:503] Master 127.0.98.62:39423 was elected leader, sending a full tablet report...
I20260929 14:49:12.359326 12099 ts_tablet_manager.cc:1428] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:49:12.359627 12080 heartbeater.cc:503] Master 127.0.98.62:39423 was elected leader, sending a full tablet report...
I20260929 14:49:12.359746 12098 tablet_bootstrap.cc:492] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: No bootstrap required, opened a new log
I20260929 14:49:12.359921 12098 ts_tablet_manager.cc:1397] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260929 14:49:12.366456 12098 raft_consensus.cc:348] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.366716 12098 raft_consensus.cc:374] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:49:12.366796 12098 raft_consensus.cc:729] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690f1578bf93499eb1e713f25e5c10c2, State: Initialized, Role: FOLLOWER
I20260929 14:49:12.367241 12098 consensus_queue.cc:260] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.367880 11808 heartbeater.cc:503] Master 127.0.98.62:39423 was elected leader, sending a full tablet report...
I20260929 14:49:12.368284 12098 ts_tablet_manager.cc:1428] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
W20260929 14:49:12.389493 11947 tablet.cc:2369] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:49:12.434518 12106 raft_consensus.cc:482] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:49:12.434818 12106 raft_consensus.cc:504] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.437075 12106 leader_election.cc:288] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f297d44038424cb09d1a623d5563f221 (127.0.98.3:45039), f714318c184243afb4cdd3aeecccf1c1 (127.0.98.2:34259)
I20260929 14:49:12.439064 12104 raft_consensus.cc:482] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:49:12.439261 12104 raft_consensus.cc:504] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.440518 12104 leader_election.cc:288] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f297d44038424cb09d1a623d5563f221 (127.0.98.3:45039), 690f1578bf93499eb1e713f25e5c10c2 (127.0.98.1:35039)
I20260929 14:49:12.441998 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.436909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50795 (local address 127.0.98.2:34259)
0929 14:49:12.437039 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.437043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.437452 (+ 409us) server_negotiation.cc:407] Connection header received
0929 14:49:12.437660 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.437663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.437714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.437764 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.438243 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.438997 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.440549 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.441192 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.441267 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.441331 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.441758 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.441847 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:49:12.442660 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.436514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34259 (local address 127.0.98.1:50795)
0929 14:49:12.437363 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.437378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.437531 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.437936 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.437939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.437968 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.438106 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.438116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.439134 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.439137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.440411 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.440423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.441509 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.441515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.441611 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.441635 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.442398 (+ 763us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.442510 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":101,"threads_started":1}
I20260929 14:49:12.443416 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "690f1578bf93499eb1e713f25e5c10c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f714318c184243afb4cdd3aeecccf1c1" is_pre_election: true
I20260929 14:49:12.443655 11900 raft_consensus.cc:2454] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690f1578bf93499eb1e713f25e5c10c2 in term 0.
I20260929 14:49:12.444253 11696 leader_election.cc:302] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 690f1578bf93499eb1e713f25e5c10c2, f714318c184243afb4cdd3aeecccf1c1; no voters:
I20260929 14:49:12.444638 12106 raft_consensus.cc:2781] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:49:12.444723 12106 raft_consensus.cc:482] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:49:12.444785 12106 raft_consensus.cc:3037] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:49:12.446907 12106 raft_consensus.cc:504] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.447957 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "690f1578bf93499eb1e713f25e5c10c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f714318c184243afb4cdd3aeecccf1c1"
I20260929 14:49:12.448096 11900 raft_consensus.cc:3037] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:49:12.449508 11900 raft_consensus.cc:2454] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690f1578bf93499eb1e713f25e5c10c2 in term 1.
I20260929 14:49:12.449950 11696 leader_election.cc:302] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 690f1578bf93499eb1e713f25e5c10c2, f714318c184243afb4cdd3aeecccf1c1; no voters:
I20260929 14:49:12.450287 12106 leader_election.cc:288] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [CANDIDATE]: Term 1 election: Requested vote from peers f297d44038424cb09d1a623d5563f221 (127.0.98.3:45039), f714318c184243afb4cdd3aeecccf1c1 (127.0.98.2:34259)
I20260929 14:49:12.450415 12106 raft_consensus.cc:2781] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:49:12.450522 12106 raft_consensus.cc:686] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [term 1 LEADER]: Becoming Leader. State: Replica: 690f1578bf93499eb1e713f25e5c10c2, State: Running, Role: LEADER
I20260929 14:49:12.450551 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.440915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45039 (local address 127.0.98.2:42705)
0929 14:49:12.444940 (+ 4025us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.444960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.445082 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.446018 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.446021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.446044 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.446193 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.446199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.447397 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.447401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.448904 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.448916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.449980 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.449984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.450130 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.450158 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.450237 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.450329 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3920,"thread_start_us":88,"threads_started":1}
I20260929 14:49:12.450941 12106 consensus_queue.cc:237] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } }
I20260929 14:49:12.451287 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.441805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42705 (local address 127.0.98.3:45039)
0929 14:49:12.445394 (+ 3589us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.445403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.445417 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:49:12.445458 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.445462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.445507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.445595 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.446318 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.447209 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.449063 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.449715 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.449824 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.449915 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.450925 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.451016 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3503,"thread_start_us":90,"threads_started":1}
I20260929 14:49:12.452495 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "f714318c184243afb4cdd3aeecccf1c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f297d44038424cb09d1a623d5563f221" is_pre_election: true
I20260929 14:49:12.452886 12032 raft_consensus.cc:2454] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f714318c184243afb4cdd3aeecccf1c1 in term 0.
I20260929 14:49:12.453430 11836 leader_election.cc:302] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f297d44038424cb09d1a623d5563f221, f714318c184243afb4cdd3aeecccf1c1; no voters:
I20260929 14:49:12.453770 12104 raft_consensus.cc:2741] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 14:49:12.454568 11595 catalog_manager.cc:5495] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 690f1578bf93499eb1e713f25e5c10c2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "690f1578bf93499eb1e713f25e5c10c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "690f1578bf93499eb1e713f25e5c10c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 } health_report { overall_health: UNKNOWN } } }
I20260929 14:49:12.460120 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.436681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37389 (local address 127.0.98.3:45039)
0929 14:49:12.436810 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.436814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.440635 (+ 3821us) server_negotiation.cc:407] Connection header received
0929 14:49:12.442765 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.442770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.442838 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.442897 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.443411 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.444326 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.458722 (+ 14396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.459355 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.459475 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.459544 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.459896 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.459981 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:49:12.460810 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.436514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45039 (local address 127.0.98.1:37389)
0929 14:49:12.438882 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.438896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.442380 (+ 3484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.443087 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.443090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.443111 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.443268 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.443275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.444513 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.444516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.458554 (+ 14038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.458568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.459619 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.459623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.459749 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.459776 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.460514 (+ 738us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.460636 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"mutex_wait_us":309,"thread_start_us":129,"threads_started":1}
I20260929 14:49:12.461663 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "690f1578bf93499eb1e713f25e5c10c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f297d44038424cb09d1a623d5563f221" is_pre_election: true
I20260929 14:49:12.461833 12032 raft_consensus.cc:2454] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 690f1578bf93499eb1e713f25e5c10c2 in term 0.
I20260929 14:49:12.462081 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "690f1578bf93499eb1e713f25e5c10c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f297d44038424cb09d1a623d5563f221"
I20260929 14:49:12.462184 12034 raft_consensus.cc:3037] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:49:12.462767 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.442029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51163 (local address 127.0.98.1:35039)
0929 14:49:12.442168 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.442173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.445890 (+ 3717us) server_negotiation.cc:407] Connection header received
0929 14:49:12.446139 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.446145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.446206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.446317 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:49:12.459155 (+ 12838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.459847 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.461389 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.461987 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.462068 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.462140 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.462550 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.462633 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:49:12.463946 12034 raft_consensus.cc:2454] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 690f1578bf93499eb1e713f25e5c10c2 in term 1.
I20260929 14:49:12.468580 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.441283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35039 (local address 127.0.98.2:51163)
0929 14:49:12.445792 (+ 4509us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.445807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:49:12.445913 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:49:12.458768 (+ 12855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:49:12.458772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:49:12.458803 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:49:12.458941 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.458949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.459997 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.460001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.461226 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:49:12.461237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.462251 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.462254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:49:12.462371 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.462399 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:49:12.468305 (+ 5906us) client_negotiation.cc:240] Negotiation successful
0929 14:49:12.468411 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4440,"thread_start_us":106,"threads_started":1}
I20260929 14:49:12.469601 11748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234550db8974442f823ce870896503ad" candidate_uuid: "f714318c184243afb4cdd3aeecccf1c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "690f1578bf93499eb1e713f25e5c10c2" is_pre_election: true
I20260929 14:49:12.474926 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.469609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.0.98.1:35039)
0929 14:49:12.469744 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:49:12.469749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:49:12.469765 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:49:12.469957 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:49:12.469960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:49:12.470016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:49:12.470122 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:49:12.470538 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.471381 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.472793 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:49:12.473159 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:49:12.473289 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:49:12.473652 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:49:12.474177 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:49:12.474306 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:49:12.474710 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 14:49:12.474783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:49:12.482404 11900 raft_consensus.cc:1261] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 1 FOLLOWER]: Refusing update from remote peer 690f1578bf93499eb1e713f25e5c10c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:49:12.483177 12106 consensus_queue.cc:1035] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f714318c184243afb4cdd3aeecccf1c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:49:12.487347 12034 raft_consensus.cc:1261] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 1 FOLLOWER]: Refusing update from remote peer 690f1578bf93499eb1e713f25e5c10c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:49:12.488071 12106 consensus_queue.cc:1035] T 234550db8974442f823ce870896503ad P 690f1578bf93499eb1e713f25e5c10c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f297d44038424cb09d1a623d5563f221" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:49:12.494856 12115 mvcc.cc:205] Tried to move back new op lower bound from 7334679971760529408 to 7334679971642593280. Current Snapshot: MvccSnapshot[applied={T|T < 7334679971760529408}]
I20260929 14:49:12.495501 12117 mvcc.cc:205] Tried to move back new op lower bound from 7334679971760529408 to 7334679971642593280. Current Snapshot: MvccSnapshot[applied={T|T < 7334679971760529408}]
I20260929 14:49:12.504320 12119 mvcc.cc:205] Tried to move back new op lower bound from 7334679971760529408 to 7334679971642593280. Current Snapshot: MvccSnapshot[applied={T|T < 7334679971760529408}]
Sep 29 14:49:12 dist-test-slave-5hd1 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:12.512315 11596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:49:12.512501 11596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:49:12.515744 11596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dff27961bb4f4d22bc0cf32253ebbcb1: Sending DeleteTablet for 3 replicas of tablet 234550db8974442f823ce870896503ad
I20260929 14:49:12.516661 11740 tablet_service.cc:1514] Processing DeleteTablet for tablet 234550db8974442f823ce870896503ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37732
I20260929 14:49:12.517542 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet 234550db8974442f823ce870896503ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56552
I20260929 14:49:12.518601 11880 tablet_service.cc:1514] Processing DeleteTablet for tablet 234550db8974442f823ce870896503ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58284
I20260929 14:49:12.519822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 11677
I20260929 14:49:12.522310 12128 tablet_replica.cc:330] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1: stopping tablet replica
I20260929 14:49:12.522760 12128 raft_consensus.cc:2229] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:49:12.522998 12127 tablet_replica.cc:330] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: stopping tablet replica
I20260929 14:49:12.523284 12127 raft_consensus.cc:2229] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:49:12.523713 12128 raft_consensus.cc:2258] T 234550db8974442f823ce870896503ad P f714318c184243afb4cdd3aeecccf1c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:49:12.524013 12127 raft_consensus.cc:2258] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:49:12.527465 12127 ts_tablet_manager.cc:1905] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:49:12.531472 12127 ts_tablet_manager.cc:1918] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:49:12.531606 12127 log.cc:1196] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/234550db8974442f823ce870896503ad
I20260929 14:49:12.531983 12127 ts_tablet_manager.cc:1939] T 234550db8974442f823ce870896503ad P f297d44038424cb09d1a623d5563f221: Deleting consensus metadata
W20260929 14:49:12.533264 11582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35039 (error 108)
I20260929 14:49:12.533902 11583 catalog_manager.cc:4841] TS f297d44038424cb09d1a623d5563f221 (127.0.98.3:45039): tablet 234550db8974442f823ce870896503ad (table test-table [id=7772a6e9c37649d68a38cfa656aead8e]) successfully deleted
I20260929 14:49:12.534271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 11816
I20260929 14:49:12.541839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 11951
I20260929 14:49:12.545141 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.544714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35039 (local address 127.0.0.1:37750)
0929 14:49:12.544947 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.545036 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260929 14:49:12.545361 11582 catalog_manager.cc:4584] TS 690f1578bf93499eb1e713f25e5c10c2 (127.0.98.1:35039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 234550db8974442f823ce870896503ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35039: connect: Connection refused (error 111)
I20260929 14:49:12.547326 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:49:12.547018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34259 (local address 127.0.0.1:58292)
0929 14:49:12.547179 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 14:49:12.547245 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 14:49:12.553895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 11564
I20260929 14:49:12.563603 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:49:12.909384 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:49:12.909524 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:49:13.254788 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:49:13.255 UTC [11150] LOG: received smart shutdown request
2026-09-29 14:49:13.259 UTC [11150] LOG: background worker "logical replication launcher" (PID 11159) exited with exit code 1
2026-09-29 14:49:13.259 UTC [11154] LOG: shutting down
2026-09-29 14:49:13.448 UTC [11150] LOG: database system is shut down
2026-09-29T14:49:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50650 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:49:13 dist-test-slave-5hd1 krb5kdc[12139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:49:13 dist-test-slave-5hd1 krb5kdc[12139](info): set up 2 sockets
Sep 29 14:49:13 dist-test-slave-5hd1 krb5kdc[12139](info): commencing operation
krb5kdc: starting...
W20260929 14:49:14.684962 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:49:14 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:49:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:49:14.887750 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiGqqz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:49:17.753 UTC [12169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:49:17.753 UTC [12169] LOG: listening on IPv4 address "127.0.98.20", port 38375
2026-09-29 14:49:17.756 UTC [12169] LOG: listening on Unix socket "/tmp/.s.PGSQL.38375"
2026-09-29 14:49:17.796 UTC [12172] LOG: database system was shut down at 2026-09-29 14:49:16 UTC
2026-09-29 14:49:17.801 UTC [12169] LOG: database system is ready to accept connections
I20260929 14:49:18.752722 392 mini_postgres.cc:96] Postgres bound to 38375
2026-09-29 14:49:18.760 UTC [12180] FATAL: database "slave" does not exist
127.0.98.20:38375 - accepting connections
I20260929 14:49:18.760790 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:49:18.778627 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:49:19.338583 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:49:19.338660 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin
2026-09-29 14:49:19,637 [I] DB FLAVOR :POSTGRES
2026-09-29 14:49:19,637 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:49:19,637 [I] Checking connection..
2026-09-29 14:49:19,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:49:19,904 [I] Checking connection passed.
2026-09-29 14:49:19,904 [I] --------- Verifying version history table ---------
2026-09-29 14:49:19,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:49:20,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:49:20,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:49:20,411 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:49:20,412 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:49:20,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:49:20.654 UTC [12244] WARNING: there is no transaction in progress
2026-09-29 14:49:20,662 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:49:20,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:49:20,929 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:49:20,929 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:49:20,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:49:21,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:49:21,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5hd1.c.gcp-upstream.internal','N') ;"
2026-09-29 14:49:21,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:49:21,957 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:49:21,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:49:22,208 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:49:22,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:49:22,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:49:22,717 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:49:22,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:49:23.784 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.801 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.819 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.836 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.853 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.873 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.945 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.952 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.968 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.980 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:23.991 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24.005 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24.017 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24.026 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24.035 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24.044 UTC [12352] WARNING: there is no transaction in progress
2026-09-29 14:49:24,840 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:49:24,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:49:25,120 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:49:25,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:49:25,404 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:49:25,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:49:25,692 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:49:25,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:49:25,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:49:26,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5hd1.c.gcp-upstream.internal';"
2026-09-29 14:49:26,492 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:49:26,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:49:26,750 [I] DB_PATCHES have already been applied
I20260929 14:49:26.757467 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiGqqz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:49:26.757543 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:49:26.761176 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39881
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:49:27.927414 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.167s user 0.003s sys 0.008s
Sep 29, 2026 2:49:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:49:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 2:49:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:49:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:49:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:49:51.801805 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:49:51 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693391, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:49:51.894196 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:49:51.905138 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:49:52.866616 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:49:52.866688 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms
I20260929 14:49:52.866700 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38375
2026-09-29 14:49:53,049 [I] DB FLAVOR :POSTGRES
2026-09-29 14:49:53,049 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:49:53,050 [I] Checking connection
2026-09-29 14:49:53,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:49:53,340 [I] connection success
2026-09-29 14:49:53,341 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:49:53,341 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:49:53,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:49:53,635 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:49:53,635 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:49:53,636 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:49:53,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38375/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:49:53,921 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:49:54.453713 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:49:54.681067 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:49:54.726347 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:49:54 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693391, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:49:55.168601 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:49:55.168874 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiGqqz/build/debug/bin/postgresql.jar:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiGqqz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:49:55.168901 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:49:55.172966 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12533
I20260929 14:49:55.173132 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60291
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:49:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:49:57.471029 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.577s user 0.012s sys 0.153s
I20260929 14:49:57.471182 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:49:57.471264 392 mini_ranger_kms.cc:401] 127.0.98.20:60291/kms/v1/keys
Sep 29 14:49:57 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:49:58 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:49:58 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:50:01 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:50:01.169348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44487
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:01.286980 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:01.287307 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:01.287375 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:01.291591 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:01.291697 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:01.291743 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:01.291776 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:01.291805 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:01.297540 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60291/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44487
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:01.298619 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:01.299671 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:50:01.309450 12583 server_base.cc:1034] running on GCE node
W20260929 14:50:01.309518 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:01.309518 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:01.309708 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:01.310649 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:01.312072 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:01.313244 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790693401313223 us; error 41 us; skew 500 ppm
Sep 29 14:50:01 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:01.316466 12583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:01.317596 12583 webserver.cc:460] Webserver started at http://127.0.98.62:33173/ using document root <none> and password file <none>
I20260929 14:50:01.318254 12583 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:01.318341 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:01.318614 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:50:01 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:01.368157 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "ea990a812bc04dae957f441e288a4ff2"
format_stamp: "Formatted at 2026-09-29 14:50:01 on dist-test-slave-5hd1"
server_key: "c6c144b283c02c4e296c568104568890"
server_key_iv: "07b77c56e107f67b5621e4621c69d689"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:01.369139 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea990a812bc04dae957f441e288a4ff2"
format_stamp: "Formatted at 2026-09-29 14:50:01 on dist-test-slave-5hd1"
server_key: "c6c144b283c02c4e296c568104568890"
server_key_iv: "07b77c56e107f67b5621e4621c69d689"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:01.374498 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 14:50:01.396275 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:01.398340 12583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 14:50:01.398541 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "ea990a812bc04dae957f441e288a4ff2"
format_stamp: "Formatted at 2026-09-29 14:50:01 on dist-test-slave-5hd1"
server_key: "c6c144b283c02c4e296c568104568890"
server_key_iv: "07b77c56e107f67b5621e4621c69d689"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:01.398726 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:01.423103 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:01.427284 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:01.427523 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:01.438592 12583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44487
I20260929 14:50:01.438612 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44487 every 8 connection(s)
I20260929 14:50:01.440160 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:01.443454 12651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:01.449455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 12583
I20260929 14:50:01.449534 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Bootstrap starting.
I20260929 14:50:01.449608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:50:01.449957 392 external_mini_cluster.cc:1428] Setting key eceb6e98a9ea066403467cab2e7ca2ba
I20260929 14:50:01.452970 12651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:01.453783 12651 log.cc:824] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:01.456074 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: No bootstrap required, opened a new log
I20260929 14:50:01.460451 12651 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:01.460773 12651 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:01.460886 12651 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea990a812bc04dae957f441e288a4ff2, State: Initialized, Role: FOLLOWER
I20260929 14:50:01.461472 12651 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:01.461655 12651 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:01.461781 12651 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:01.461911 12651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 14:50:01 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:01.463351 12651 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:01.463753 12651 leader_election.cc:302] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea990a812bc04dae957f441e288a4ff2; no voters:
I20260929 14:50:01.464242 12651 leader_election.cc:288] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:50:01.464376 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:01.464632 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 LEADER]: Becoming Leader. State: Replica: ea990a812bc04dae957f441e288a4ff2, State: Running, Role: LEADER
I20260929 14:50:01.464983 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:01.465692 12651 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:01.466285 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea990a812bc04dae957f441e288a4ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } } }
I20260929 14:50:01.466634 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:01.466935 12656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea990a812bc04dae957f441e288a4ff2. Latest consensus state: current_term: 1 leader_uuid: "ea990a812bc04dae957f441e288a4ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } } }
I20260929 14:50:01.467458 12656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:01.468128 12665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:01.468326 12651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:50:01.470978 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:01.451447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.0.98.62:44487)
0929 14:50:01.452074 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:01.452102 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:01.452166 (+ 64us) server_negotiation.cc:407] Connection header received
0929 14:50:01.453318 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:01.453351 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:01.453926 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:01.454440 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:01.455513 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:01.457311 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:01.458795 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:01.459599 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:01.459784 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:01.463328 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:01.463336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:01.463359 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:01.463399 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:01.466204 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:01.466896 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:01.466903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:01.466910 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:01.467162 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:01.467545 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:01.467549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:01.467551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:01.468213 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:01.468421 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:01.469419 (+ 998us) server_negotiation.cc:299] Negotiation successful
0929 14:50:01.469780 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":136,"threads_started":1}
I20260929 14:50:01.476043 12665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:01.487008 12665 catalog_manager.cc:1347] Generated new cluster ID: c40cfd81b4d842d1b46b164915572bee
I20260929 14:50:01.487229 12665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:01.506855 12665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:50:01.507939 12665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:01.512703 12665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Generated new TSK 0
I20260929 14:50:01.513761 12665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:50:02.256366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:02.410504 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:02.410764 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:02.410807 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:02.415318 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:02.415392 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:02.415454 12696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:02.422714 12696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:02.424070 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:02.425376 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:02.436731 12706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:02.436731 12704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:02.436841 12703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:02.437150 12696 server_base.cc:1034] running on GCE node
I20260929 14:50:02.437737 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:02.438567 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:02.439733 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790693402439666 us; error 91 us; skew 500 ppm
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:02.443987 12696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:02.445286 12696 webserver.cc:460] Webserver started at http://127.0.98.1:44197/ using document root <none> and password file <none>
I20260929 14:50:02.446097 12696 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:02.446255 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:02.446565 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:02.465530 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ae1ad91dec74820a8a6ad84b95050ed"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "20c72877a10139b18a4167ff07006b2b"
server_key_iv: "7c097d7df40d8ecaca471a545400d005"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.466246 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ae1ad91dec74820a8a6ad84b95050ed"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "20c72877a10139b18a4167ff07006b2b"
server_key_iv: "7c097d7df40d8ecaca471a545400d005"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.471009 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:50:02.496512 12713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.498334 12696 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20260929 14:50:02.498575 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "2ae1ad91dec74820a8a6ad84b95050ed"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "20c72877a10139b18a4167ff07006b2b"
server_key_iv: "7c097d7df40d8ecaca471a545400d005"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.498812 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:02.516640 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:02.520680 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:02.520970 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:02.521814 12696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:02.522985 12696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:02.523098 12696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.523233 12696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:02.523319 12696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.535166 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41873
I20260929 14:50:02.536432 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:02.540567 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41873 every 8 connection(s)
I20260929 14:50:02.542187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 12696
I20260929 14:50:02.542323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:50:02.542670 392 external_mini_cluster.cc:1428] Setting key 0aed025d8b2b139ba06b4dd52d2a4101
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:02.567087 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:02.546423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60195 (local address 127.0.98.62:44487)
0929 14:50:02.550645 (+ 4222us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:02.550650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:02.550666 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:50:02.550714 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:02.550718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:02.550773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:02.550871 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:02.552582 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.553436 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.555144 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.555470 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.555572 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:02.561485 (+ 5913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:02.561489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:02.561492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:02.561555 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:02.563495 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:02.564143 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:02.564146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:02.564148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:02.564264 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:02.564686 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:02.564689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:02.564691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:02.564927 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:02.565027 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:02.566712 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0929 14:50:02.566851 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4133,"thread_start_us":109,"threads_started":1}
I20260929 14:50:02.568922 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:02.545027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.1:60195)
0929 14:50:02.548863 (+ 3836us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:02.548931 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:02.550175 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:02.551214 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:02.551224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:02.551619 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:02.552388 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:02.552411 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.553588 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.553592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:02.554987 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:02.554999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.555760 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.555764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:02.555981 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:02.558232 (+ 2251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:02.558254 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:02.561243 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:02.563648 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:02.563654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:02.563673 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:02.564023 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:02.564368 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:02.564371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:02.564373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:02.564568 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:02.565084 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:02.565091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:02.565437 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 14:50:02.567318 (+ 1881us) client_negotiation.cc:240] Negotiation successful
0929 14:50:02.567763 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3580,"thread_start_us":164,"threads_started":1}
I20260929 14:50:02.570703 12829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:02.570959 12829 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:02.571558 12829 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:02.573338 12615 ts_manager.cc:194] Registered new tserver with Master: 2ae1ad91dec74820a8a6ad84b95050ed (127.0.98.1:41873)
I20260929 14:50:02.575018 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60195
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:50:02.604015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:50:02.772601 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:02.773368 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:02.773495 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:02.779603 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:02.779810 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:02.779944 12835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:02.787737 12835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:02.789285 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:02.790602 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:02.804283 12842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:02.804320 12841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:02.804297 12844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:02.805871 12835 server_base.cc:1034] running on GCE node
I20260929 14:50:02.806404 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:02.807211 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:02.810537 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790693402810430 us; error 125 us; skew 500 ppm
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:02.815181 12835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:02.816784 12835 webserver.cc:460] Webserver started at http://127.0.98.2:37807/ using document root <none> and password file <none>
I20260929 14:50:02.817771 12835 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:02.817905 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:02.818246 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:02.837319 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f3b749d236b4f36838b6dcd815c511e"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "87703e6d3b449f1e7343b480773b1d83"
server_key_iv: "8cacdc3dc464d30da46d221af2e7281a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.838364 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f3b749d236b4f36838b6dcd815c511e"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "87703e6d3b449f1e7343b480773b1d83"
server_key_iv: "8cacdc3dc464d30da46d221af2e7281a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.847005 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 14:50:02.882398 12851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.884783 12835 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.000s
I20260929 14:50:02.884954 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "5f3b749d236b4f36838b6dcd815c511e"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "87703e6d3b449f1e7343b480773b1d83"
server_key_iv: "8cacdc3dc464d30da46d221af2e7281a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:02.885162 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:02.907656 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:02.912231 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:02.912508 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:02.913427 12835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:02.916217 12835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:02.916328 12835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.916435 12835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:02.916500 12835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:02.932062 12835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34739
I20260929 14:50:02.933622 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:02.934806 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34739 every 8 connection(s)
I20260929 14:50:02.939714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 12835
I20260929 14:50:02.939875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:50:02.940354 392 external_mini_cluster.cc:1428] Setting key ad5a1447116eb53459699eaa5d1137a9
Sep 29 14:50:02 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:02.965466 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:02.938061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47257 (local address 127.0.98.62:44487)
0929 14:50:02.938201 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:02.938207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:02.949228 (+ 11021us) server_negotiation.cc:407] Connection header received
0929 14:50:02.950665 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:02.950668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:02.950729 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:02.950811 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:02.952424 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.952998 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.954607 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.954846 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.954962 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:02.958117 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:02.958123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:02.958127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:02.958177 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:02.960395 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:02.961236 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:02.961240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:02.961243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:02.961340 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:02.961814 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:02.961820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:02.961825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:02.962071 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:02.962154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:02.965070 (+ 2916us) server_negotiation.cc:299] Negotiation successful
0929 14:50:02.965243 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:50:02.967893 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:02.938144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.2:47257)
0929 14:50:02.948987 (+ 10843us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:02.949089 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:02.950387 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:02.951022 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:02.951033 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:02.951480 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:02.952274 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:02.952298 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.953135 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.953139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:02.954491 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:02.954502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:02.955010 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:02.955015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:02.955223 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:02.955946 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:02.955968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:02.957823 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:02.960598 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:02.960610 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:02.960638 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:02.961104 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:02.961455 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:02.961460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:02.961462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:02.961677 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:02.962241 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:02.962249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:02.962640 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 14:50:02.966075 (+ 3435us) client_negotiation.cc:240] Negotiation successful
0929 14:50:02.966564 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10486,"thread_start_us":194,"threads_started":1}
I20260929 14:50:02.970973 12965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:02.971366 12965 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:02.972375 12965 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:02.974005 12615 ts_manager.cc:194] Registered new tserver with Master: 5f3b749d236b4f36838b6dcd815c511e (127.0.98.2:34739)
I20260929 14:50:02.974759 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47257
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:50:03.012457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:03.170322 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:03.170598 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:03.170673 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:03.174685 12970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:03.174770 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:03.174877 12970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:03.181493 12970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:03.182554 12970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:03.183750 12970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:03.194851 12976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.195255 12975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.195338 12978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:03.196156 12970 server_base.cc:1034] running on GCE node
I20260929 14:50:03.196736 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:03.197535 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:03.200048 12970 hybrid_clock.cc:648] HybridClock initialized: now 1790693403199971 us; error 98 us; skew 500 ppm
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:03.203891 12970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:03.205145 12970 webserver.cc:460] Webserver started at http://127.0.98.3:35789/ using document root <none> and password file <none>
I20260929 14:50:03.205860 12970 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:03.205958 12970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:03.206194 12970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:03.238905 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "1af03e54bd234d0894692030c6555b06"
format_stamp: "Formatted at 2026-09-29 14:50:03 on dist-test-slave-5hd1"
server_key: "9b8cac00ca42b6296824c19f20b44953"
server_key_iv: "c963c71877db109e352be9e0aa8c680a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:03.239766 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1af03e54bd234d0894692030c6555b06"
format_stamp: "Formatted at 2026-09-29 14:50:03 on dist-test-slave-5hd1"
server_key: "9b8cac00ca42b6296824c19f20b44953"
server_key_iv: "c963c71877db109e352be9e0aa8c680a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:03.248052 12970 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 14:50:03.292503 12985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.297386 12970 fs_manager.cc:725] Time spent opening block manager: real 0.031s user 0.002s sys 0.000s
I20260929 14:50:03.297551 12970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "1af03e54bd234d0894692030c6555b06"
format_stamp: "Formatted at 2026-09-29 14:50:03 on dist-test-slave-5hd1"
server_key: "9b8cac00ca42b6296824c19f20b44953"
server_key_iv: "c963c71877db109e352be9e0aa8c680a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:03.297710 12970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:03.324895 12970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:03.330681 12970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:03.330927 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:03.331728 12970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:03.332989 12970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:03.333055 12970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.333109 12970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:03.333144 12970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.346177 12970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38631
I20260929 14:50:03.346378 13098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38631 every 8 connection(s)
I20260929 14:50:03.347487 12970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:03.356884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 12970
I20260929 14:50:03.357048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:50:03.357451 392 external_mini_cluster.cc:1428] Setting key b1a6862ae0689c03420eebb50a9e6379
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:03.365001 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:03.349992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36131 (local address 127.0.98.62:44487)
0929 14:50:03.350231 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:03.350236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:03.351483 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 14:50:03.352840 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:03.352843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:03.352917 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:03.353030 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:03.354515 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:03.355104 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:03.356875 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:03.357237 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:03.357368 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:03.360778 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:03.360783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:03.360786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:03.360817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:03.362485 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:03.363104 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:03.363109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:03.363111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:03.363192 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:03.363570 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:03.363573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:03.363575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:03.363796 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:03.363922 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:03.364646 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 14:50:03.364804 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 14:50:03.366992 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:03.350640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.3:36131)
0929 14:50:03.351258 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:03.351325 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:03.352503 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:03.353259 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:03.353268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:03.353635 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:03.354315 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:03.354334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:03.355262 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:03.355266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:03.356712 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:03.356723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:03.357424 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:03.357428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:03.357632 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:03.358541 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:03.358566 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:03.360589 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:03.362626 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:03.362634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:03.362654 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:03.362984 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:03.363291 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:03.363294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:03.363296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:03.363461 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:03.363917 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:03.363925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:03.364342 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 14:50:03.365307 (+ 965us) client_negotiation.cc:240] Negotiation successful
0929 14:50:03.365712 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":140,"threads_started":1}
I20260929 14:50:03.368505 13099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:03.368872 13099 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:03.369565 13099 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:03.370941 12615 ts_manager.cc:194] Registered new tserver with Master: 1af03e54bd234d0894692030c6555b06 (127.0.98.3:38631)
I20260929 14:50:03.371740 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36131
I20260929 14:50:03.372279 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:03.373024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 12696
I20260929 14:50:03.381145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 12835
I20260929 14:50:03.387476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 12970
I20260929 14:50:03.393589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 12583
I20260929 14:50:03.401986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44487
--webserver_interface=127.0.98.62
--webserver_port=33173
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44487
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:03.537850 13102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:03.538215 13102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:03.538259 13102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:03.543287 13102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:03.543403 13102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:03.543435 13102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:03.543454 13102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:03.543471 13102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:03.551674 13102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44487
--ranger_config_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:03.553159 13102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:03.554468 13102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:03.564867 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.564816 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.564816 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:03.565080 13102 server_base.cc:1034] running on GCE node
I20260929 14:50:03.565955 13102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:03.567137 13102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:03.568285 13102 hybrid_clock.cc:648] HybridClock initialized: now 1790693403568261 us; error 38 us; skew 500 ppm
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:03.571437 13102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:03.572413 13102 webserver.cc:460] Webserver started at http://127.0.98.62:33173/ using document root <none> and password file <none>
I20260929 14:50:03.572954 13102 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:03.573019 13102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:03.645888 13102 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:50:03.649019 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.650851 13102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:50:03.651047 13102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "ea990a812bc04dae957f441e288a4ff2"
format_stamp: "Formatted at 2026-09-29 14:50:01 on dist-test-slave-5hd1"
server_key: "c6c144b283c02c4e296c568104568890"
server_key_iv: "07b77c56e107f67b5621e4621c69d689"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:03.651775 13102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:03.667601 13102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:03.671497 13102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:03.671789 13102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:03.680965 13102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44487
I20260929 14:50:03.680977 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44487 every 8 connection(s)
I20260929 14:50:03.682330 13102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:03.682828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13102
I20260929 14:50:03.683349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41873
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44487
--webserver_port=44197
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:03.687289 13171 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:50:03.689093 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Bootstrap starting.
I20260929 14:50:03.706476 13171 log.cc:824] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:03.713176 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:03.713598 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Bootstrap complete.
I20260929 14:50:03.717819 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:03.718241 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea990a812bc04dae957f441e288a4ff2, State: Initialized, Role: FOLLOWER
I20260929 14:50:03.718753 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:03.718905 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:03.719005 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:03.719132 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:50:03.721276 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:03.721690 13171 leader_election.cc:302] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea990a812bc04dae957f441e288a4ff2; no voters:
I20260929 14:50:03.722043 13171 leader_election.cc:288] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:50:03.722381 13175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:50:03.722854 13175 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [term 2 LEADER]: Becoming Leader. State: Replica: ea990a812bc04dae957f441e288a4ff2, State: Running, Role: LEADER
I20260929 14:50:03.723255 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } }
I20260929 14:50:03.723563 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:03.725028 13176 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea990a812bc04dae957f441e288a4ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } } }
I20260929 14:50:03.725185 13176 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:03.725982 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:03.726174 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea990a812bc04dae957f441e288a4ff2. Latest consensus state: current_term: 2 leader_uuid: "ea990a812bc04dae957f441e288a4ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea990a812bc04dae957f441e288a4ff2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44487 } } }
I20260929 14:50:03.726289 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:03.726369 13171 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:50:03.733726 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:03.734731 13186 catalog_manager.cc:1259] Loaded cluster ID: c40cfd81b4d842d1b46b164915572bee
I20260929 14:50:03.734828 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:03.737695 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:03.738611 13186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Loaded TSK: 0
I20260929 14:50:03.740298 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:50:03.838946 13173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:03.839440 13173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:03.839524 13173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:03.846688 13173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:03.846822 13173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:03.846927 13173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:03.856630 13173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44197
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:03.858693 13173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:03.860492 13173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:03.876583 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.879691 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:03.879702 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:03.880574 13173 server_base.cc:1034] running on GCE node
I20260929 14:50:03.881146 13173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:03.882046 13173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:03.883354 13173 hybrid_clock.cc:648] HybridClock initialized: now 1790693403883319 us; error 56 us; skew 500 ppm
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:03.887401 13173 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:03.888693 13173 webserver.cc:460] Webserver started at http://127.0.98.1:44197/ using document root <none> and password file <none>
I20260929 14:50:03.889468 13173 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:03.889582 13173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:50:03 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:03.925525 13173 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:50:03.929220 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.931231 13173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 14:50:03.931452 13173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "2ae1ad91dec74820a8a6ad84b95050ed"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "20c72877a10139b18a4167ff07006b2b"
server_key_iv: "7c097d7df40d8ecaca471a545400d005"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:03.932397 13173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:03.961392 13173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:03.969839 13173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:03.970197 13173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:03.971282 13173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:03.972723 13173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:03.972846 13173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 14:50:03.973006 13173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:03.973101 13173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:03.987471 13173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41873
I20260929 14:50:03.987756 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41873 every 8 connection(s)
I20260929 14:50:03.989424 13173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:03.995219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13173
I20260929 14:50:03.995760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34739
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44487
--webserver_port=37807
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:04.020413 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:03.993406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.1:39717)
0929 14:50:03.994165 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:03.994307 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:03.996235 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:03.999432 (+ 3197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:03.999443 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:03.999894 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:04.000735 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.000758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.002870 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.002875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.004379 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.004397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.005497 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.005501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.005704 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.006580 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:04.006603 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:04.011605 (+ 5002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:04.015392 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.015403 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.015428 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.015902 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.016483 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.016490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.016494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.016709 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.017724 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:04.017754 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:04.018187 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 14:50:04.018446 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 14:50:04.018942 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":202,"threads_started":1}
I20260929 14:50:04.022676 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:03.995280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39717 (local address 127.0.98.62:44487)
0929 14:50:03.997826 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:03.997855 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:03.997911 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:50:03.998280 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:03.998296 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:03.998864 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:04.000461 (+ 1597us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:04.000939 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.002612 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.004563 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.005300 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.006131 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.012014 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:04.012026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:04.012054 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:04.012109 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:04.015159 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.016091 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.016102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.016111 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.016334 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.016853 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.016858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.016863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.017566 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:04.021269 (+ 3703us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:04.021446 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 14:50:04.021770 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":182,"threads_started":1}
I20260929 14:50:04.025827 13333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:04.026181 13333 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.026903 13333 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
W20260929 14:50:04.030100 13333 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44487 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:50:04.030308 13333 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.030583 13333 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:04.031631 13333 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.031836 13333 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
W20260929 14:50:04.032644 13333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:50:04.172823 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:04.173327 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:04.173410 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:04.179054 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:04.179128 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:04.179198 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:04.186666 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37807
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:04.188395 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:04.189916 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:04.202957 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:04.203094 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:04.202978 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:04.204074 13337 server_base.cc:1034] running on GCE node
I20260929 14:50:04.204638 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:04.205485 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:04.209709 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790693404209670 us; error 56 us; skew 500 ppm
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:04.213209 13337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:04.214507 13337 webserver.cc:460] Webserver started at http://127.0.98.2:37807/ using document root <none> and password file <none>
I20260929 14:50:04.215205 13337 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:04.215324 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:04.242062 13337 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 14:50:04.249306 13354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.251446 13337 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260929 14:50:04.251609 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "5f3b749d236b4f36838b6dcd815c511e"
format_stamp: "Formatted at 2026-09-29 14:50:02 on dist-test-slave-5hd1"
server_key: "87703e6d3b449f1e7343b480773b1d83"
server_key_iv: "8cacdc3dc464d30da46d221af2e7281a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:04.252511 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:04.272363 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:04.276469 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:04.276741 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:04.277643 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:04.278791 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:04.278896 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.279006 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:04.279075 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.296581 13337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34739
I20260929 14:50:04.296614 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34739 every 8 connection(s)
I20260929 14:50:04.297878 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:04.306440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13337
I20260929 14:50:04.306969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38631
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44487
--webserver_port=35789
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:04.323587 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:04.300632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49619 (local address 127.0.98.62:44487)
0929 14:50:04.300796 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:04.300801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:04.301876 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 14:50:04.303501 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:04.303506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:04.303585 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:04.303715 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:04.305305 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.306184 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.308144 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.308542 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.308633 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.312046 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:04.312052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:04.312055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:04.312090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:04.317099 (+ 5009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.321289 (+ 4190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.321294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.321295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.321444 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.321953 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.322050 (+ 97us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.322053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.322357 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:04.322511 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:04.323198 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 14:50:04.323384 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:50:04.324987 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:04.301031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.2:49619)
0929 14:50:04.301675 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:04.301749 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:04.303166 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:04.303962 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:04.303973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:04.304392 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:04.305105 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.305140 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.306417 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.306422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.307975 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.307992 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.308905 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.308909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.309134 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.309988 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:04.310011 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:04.311806 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:04.320263 (+ 8457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.320273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.320295 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.321084 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.321578 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.321582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.321584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.321818 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.322510 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:04.322520 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:04.322905 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 14:50:04.323160 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 14:50:04.323611 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":127,"threads_started":1}
I20260929 14:50:04.326752 13468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:04.327049 13468 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.327772 13468 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
W20260929 14:50:04.329955 13468 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44487 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:50:04.330114 13468 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.330331 13468 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:04.331195 13468 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.331393 13468 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
W20260929 14:50:04.332237 13468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:50:04.448240 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:04.448586 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:04.448690 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:04.454072 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:04.454226 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:04.454375 13471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:04.461390 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35789
--tserver_master_addrs=127.0.98.62:44487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:04.462648 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:04.464118 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:04.476526 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:04.476713 13471 server_base.cc:1034] running on GCE node
W20260929 14:50:04.476837 13476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:04.480376 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:04.481017 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:04.481853 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:04.483155 13471 hybrid_clock.cc:648] HybridClock initialized: now 1790693404483061 us; error 92 us; skew 500 ppm
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:04.493878 13471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:04.495432 13471 webserver.cc:460] Webserver started at http://127.0.98.3:35789/ using document root <none> and password file <none>
I20260929 14:50:04.496392 13471 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:04.496579 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:04.538602 13471 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.001s
I20260929 14:50:04.542948 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.545140 13471 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 14:50:04.545387 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "1af03e54bd234d0894692030c6555b06"
format_stamp: "Formatted at 2026-09-29 14:50:03 on dist-test-slave-5hd1"
server_key: "9b8cac00ca42b6296824c19f20b44953"
server_key_iv: "c963c71877db109e352be9e0aa8c680a"
server_key_version: "kuduclusterkey@0"
I20260929 14:50:04.546187 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:04.581378 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:04.585121 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:04.585363 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:04.586019 13471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:04.586939 13471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:04.587029 13471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.587106 13471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:04.587167 13471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:04.599721 13471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38631
I20260929 14:50:04.599812 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38631 every 8 connection(s)
I20260929 14:50:04.601192 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:04.609802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13471
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:04.619796 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:04.603862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58839 (local address 127.0.98.62:44487)
0929 14:50:04.604081 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:04.604086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:04.605271 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 14:50:04.606744 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:04.606748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:04.606817 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:04.606931 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:04.608675 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.609442 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.611269 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.611662 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.611781 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.615119 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:04.615123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:04.615126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:04.615158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:04.617069 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.617893 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.617898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.617900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.618017 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.618498 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.618502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.618504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.618723 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:04.618849 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:04.619436 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 14:50:04.619585 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 14:50:04.621150 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:04.604360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44487 (local address 127.0.98.3:58839)
0929 14:50:04.605047 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:04.605121 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:04.606451 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:04.607197 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:04.607209 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:04.607630 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:04.608479 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.608503 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.609611 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.609615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.611118 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:04.611130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.611847 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.611854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:04.612080 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.612922 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:04.612948 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:04.614909 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:04.617252 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.617261 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.617280 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.617729 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.618140 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:04.618145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:04.618151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:04.618369 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:04.618848 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:04.618857 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:04.619164 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 14:50:04.619379 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 14:50:04.619738 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":163,"threads_started":1}
I20260929 14:50:04.622551 13608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44487
I20260929 14:50:04.622838 13608 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:04.623368 13608 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:04.624958 13135 ts_manager.cc:194] Registered new tserver with Master: 1af03e54bd234d0894692030c6555b06 (127.0.98.3:38631)
I20260929 14:50:04.625432 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:04.610507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.0.98.62:44487)
0929 14:50:04.611913 (+ 1406us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:04.611922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:04.611938 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:50:04.611986 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:04.611989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:04.612047 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:04.612153 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:04.613167 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.614071 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.615324 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:04.616341 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:04.616453 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:04.619471 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:04.619475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:04.619478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:04.619508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:04.621570 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.622351 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.622354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.622356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.622459 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:04.624214 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:04.624218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:04.624220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:04.624480 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:04.624733 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:04.625020 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 14:50:04.625178 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":95,"threads_started":1}
I20260929 14:50:04.626403 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58839
Sep 29 14:50:04 dist-test-slave-5hd1 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:05.032965 13333 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:05.033406 13333 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:05.034621 13135 ts_manager.cc:194] Registered new tserver with Master: 2ae1ad91dec74820a8a6ad84b95050ed (127.0.98.1:41873)
I20260929 14:50:05.035597 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39717
I20260929 14:50:05.332499 13468 heartbeater.cc:464] Registering TS with master...
Sep 29 14:50:05 dist-test-slave-5hd1 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:50:05.332877 13468 heartbeater.cc:511] Master 127.0.98.62:44487 requested a full tablet report, sending...
I20260929 14:50:05.333967 13135 ts_manager.cc:194] Registered new tserver with Master: 5f3b749d236b4f36838b6dcd815c511e (127.0.98.2:34739)
I20260929 14:50:05.334892 13135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49619
I20260929 14:50:05.340440 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:05.358511 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.342675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.0.98.62:44487)
0929 14:50:05.343537 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.343540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.343604 (+ 64us) server_negotiation.cc:407] Connection header received
0929 14:50:05.343653 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.343656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.343706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.343793 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:05.344828 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.345542 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.347839 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.348105 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.348235 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.349934 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:05.349938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:05.349941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:05.349965 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:05.352033 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:05.352567 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:05.352572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:05.352574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:05.352657 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:05.352968 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:05.352972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:05.352973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:05.356760 (+ 3787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:05.356933 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.358134 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.358269 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":121,"threads_started":1}
I20260929 14:50:05.363547 13135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:50:05.368510 13135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:50:05.398312 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.387882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.0.1:55874)
0929 14:50:05.390092 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.390155 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.390303 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.392499 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:05.392502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:05.392520 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:05.392633 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.392640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.394162 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.394166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.396322 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.396334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.397673 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.397677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.397804 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.397844 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:50:05.397963 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:50:05.398108 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2053,"mutex_wait_us":640,"thread_start_us":151,"threads_started":1}
I20260929 14:50:05.398697 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.389327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.0.98.2:34739)
0929 14:50:05.390929 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.390938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.390965 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:50:05.391056 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.391062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.391314 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.391478 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:05.392778 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.394005 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.396749 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.397650 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.397761 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.397932 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.398400 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.398501 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1476,"thread_start_us":84,"threads_started":1}
I20260929 14:50:05.399572 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.389090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.0.98.3:38631)
0929 14:50:05.391820 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.391827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.391849 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:50:05.391924 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.391928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.392100 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.392358 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:05.392773 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.394028 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.396462 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.397338 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.397441 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.397609 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.399299 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.399417 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2613,"thread_start_us":139,"threads_started":1}
I20260929 14:50:05.400012 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.389151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34739 (local address 127.0.0.1:36836)
0929 14:50:05.390568 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.390581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.390654 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.391533 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:05.391539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:05.391574 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:05.391761 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.391770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.394155 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.394158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.396592 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.396604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.398002 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.398006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.398116 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.398144 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:50:05.398218 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 14:50:05.398312 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"mutex_wait_us":83,"thread_start_us":116,"threads_started":1}
I20260929 14:50:05.404011 13402 tablet_service.cc:1467] Processing CreateTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc (DEFAULT_TABLE table=test-table [id=00d8c43c72d447aa9bebf693c9c237ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:05.404316 13536 tablet_service.cc:1467] Processing CreateTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc (DEFAULT_TABLE table=test-table [id=00d8c43c72d447aa9bebf693c9c237ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:05.405766 13536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e11bbef08f417cbd87b8afee52b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:05.406327 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e11bbef08f417cbd87b8afee52b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:05.416818 13634 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Bootstrap starting.
I20260929 14:50:05.420145 13634 tablet_bootstrap.cc:654] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:05.420674 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.387869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41873 (local address 127.0.0.1:45022)
0929 14:50:05.390093 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.390156 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.390301 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.412460 (+ 22159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:05.412465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:05.412503 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:05.412665 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.412674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.416311 (+ 3637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.416317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.417620 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.417634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.420093 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.420097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.420240 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.420278 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 14:50:05.420335 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:50:05.420475 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"thread_start_us":166,"threads_started":1}
I20260929 14:50:05.421099 13634 log.cc:824] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:05.421880 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.389501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.0.98.1:41873)
0929 14:50:05.394768 (+ 5267us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.394775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.394799 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:50:05.394877 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.394882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.395053 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.412895 (+ 17842us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:05.413081 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.414291 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.418942 (+ 4651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.419933 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.421338 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.421532 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.421592 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.421714 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5142,"thread_start_us":125,"threads_started":1}
I20260929 14:50:05.423359 13634 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: No bootstrap required, opened a new log
I20260929 14:50:05.423568 13634 ts_tablet_manager.cc:1397] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260929 14:50:05.428781 13634 raft_consensus.cc:348] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.429117 13634 raft_consensus.cc:374] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:05.429195 13634 raft_consensus.cc:729] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3b749d236b4f36838b6dcd815c511e, State: Initialized, Role: FOLLOWER
I20260929 14:50:05.428508 13266 tablet_service.cc:1467] Processing CreateTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc (DEFAULT_TABLE table=test-table [id=00d8c43c72d447aa9bebf693c9c237ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:05.429742 13634 consensus_queue.cc:260] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.430203 13266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e11bbef08f417cbd87b8afee52b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:05.430850 13634 ts_tablet_manager.cc:1428] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:05.431257 13468 heartbeater.cc:503] Master 127.0.98.62:44487 was elected leader, sending a full tablet report...
I20260929 14:50:05.438282 13636 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Bootstrap starting.
I20260929 14:50:05.441210 13636 tablet_bootstrap.cc:654] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:05.442284 13636 log.cc:824] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:05.444792 13636 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: No bootstrap required, opened a new log
I20260929 14:50:05.444998 13636 ts_tablet_manager.cc:1397] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:05.445057 13639 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Bootstrap starting.
I20260929 14:50:05.447793 13639 tablet_bootstrap.cc:654] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:05.448776 13639 log.cc:824] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:05.450026 13636 raft_consensus.cc:348] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.450343 13636 raft_consensus.cc:374] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:05.450404 13636 raft_consensus.cc:729] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af03e54bd234d0894692030c6555b06, State: Initialized, Role: FOLLOWER
I20260929 14:50:05.450881 13636 consensus_queue.cc:260] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.451278 13639 tablet_bootstrap.cc:492] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: No bootstrap required, opened a new log
I20260929 14:50:05.451498 13639 ts_tablet_manager.cc:1397] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:05.452006 13636 ts_tablet_manager.cc:1428] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:05.452478 13608 heartbeater.cc:503] Master 127.0.98.62:44487 was elected leader, sending a full tablet report...
I20260929 14:50:05.456676 13639 raft_consensus.cc:348] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.457015 13639 raft_consensus.cc:374] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:05.457194 13639 raft_consensus.cc:729] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae1ad91dec74820a8a6ad84b95050ed, State: Initialized, Role: FOLLOWER
I20260929 14:50:05.457862 13639 consensus_queue.cc:260] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.458845 13333 heartbeater.cc:503] Master 127.0.98.62:44487 was elected leader, sending a full tablet report...
I20260929 14:50:05.459424 13639 ts_tablet_manager.cc:1428] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 14:50:05.480862 13637 raft_consensus.cc:482] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:05.481190 13637 raft_consensus.cc:504] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.483044 13637 leader_election.cc:288] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1af03e54bd234d0894692030c6555b06 (127.0.98.3:38631), 2ae1ad91dec74820a8a6ad84b95050ed (127.0.98.1:41873)
I20260929 14:50:05.488854 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.483822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52307 (local address 127.0.98.3:38631)
0929 14:50:05.483962 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.483967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.484273 (+ 306us) server_negotiation.cc:407] Connection header received
0929 14:50:05.484479 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.484482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.484532 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.484578 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:05.485028 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.485834 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.487498 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.488101 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.488216 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.488288 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.488632 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.488712 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:50:05.489426 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.483330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.98.2:52307)
0929 14:50:05.484148 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.484163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.484356 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.484738 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:05.484740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:05.484757 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:05.484908 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.484914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.485976 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.485979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.487356 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.487369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.488379 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.488383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.488485 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.488507 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:50:05.489176 (+ 669us) client_negotiation.cc:240] Negotiation successful
0929 14:50:05.489265 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":4,"thread_start_us":158,"threads_started":1}
I20260929 14:50:05.489820 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.484249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37519 (local address 127.0.98.1:41873)
0929 14:50:05.484400 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.484406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.484760 (+ 354us) server_negotiation.cc:407] Connection header received
0929 14:50:05.485001 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.485005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.485072 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.485131 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:05.485602 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.486428 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.488087 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.488781 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.488867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.488946 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.489537 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.489645 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:50:05.490337 13544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e11bbef08f417cbd87b8afee52b0bc" candidate_uuid: "5f3b749d236b4f36838b6dcd815c511e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af03e54bd234d0894692030c6555b06" is_pre_election: true
I20260929 14:50:05.490427 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.483557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41873 (local address 127.0.98.2:37519)
0929 14:50:05.484630 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.484647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.484859 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.485296 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:05.485300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:05.485316 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:05.485465 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.485474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.486582 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.486586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.487942 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:05.487953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.489264 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.489269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:05.489372 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.489401 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:50:05.490162 (+ 761us) client_negotiation.cc:240] Negotiation successful
0929 14:50:05.490253 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":39,"thread_start_us":153,"threads_started":1}
I20260929 14:50:05.490693 13544 raft_consensus.cc:2454] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3b749d236b4f36838b6dcd815c511e in term 0.
I20260929 14:50:05.491158 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e11bbef08f417cbd87b8afee52b0bc" candidate_uuid: "5f3b749d236b4f36838b6dcd815c511e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" is_pre_election: true
I20260929 14:50:05.491528 13285 raft_consensus.cc:2454] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3b749d236b4f36838b6dcd815c511e in term 0.
I20260929 14:50:05.492141 13356 leader_election.cc:302] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1af03e54bd234d0894692030c6555b06, 2ae1ad91dec74820a8a6ad84b95050ed, 5f3b749d236b4f36838b6dcd815c511e; no voters:
I20260929 14:50:05.492555 13637 raft_consensus.cc:2781] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:05.492643 13637 raft_consensus.cc:482] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:05.492688 13637 raft_consensus.cc:3037] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:50:05.493825 13334 tablet.cc:2369] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:50:05.494359 13637 raft_consensus.cc:504] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.494856 13637 leader_election.cc:288] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [CANDIDATE]: Term 1 election: Requested vote from peers 1af03e54bd234d0894692030c6555b06 (127.0.98.3:38631), 2ae1ad91dec74820a8a6ad84b95050ed (127.0.98.1:41873)
I20260929 14:50:05.495246 13544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e11bbef08f417cbd87b8afee52b0bc" candidate_uuid: "5f3b749d236b4f36838b6dcd815c511e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af03e54bd234d0894692030c6555b06"
I20260929 14:50:05.495339 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e11bbef08f417cbd87b8afee52b0bc" candidate_uuid: "5f3b749d236b4f36838b6dcd815c511e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae1ad91dec74820a8a6ad84b95050ed"
I20260929 14:50:05.495371 13544 raft_consensus.cc:3037] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:05.495472 13285 raft_consensus.cc:3037] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:05.496881 13544 raft_consensus.cc:2454] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3b749d236b4f36838b6dcd815c511e in term 1.
I20260929 14:50:05.497078 13285 raft_consensus.cc:2454] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3b749d236b4f36838b6dcd815c511e in term 1.
I20260929 14:50:05.497291 13355 leader_election.cc:302] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af03e54bd234d0894692030c6555b06, 5f3b749d236b4f36838b6dcd815c511e; no voters:
I20260929 14:50:05.497583 13637 raft_consensus.cc:2781] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:05.497848 13637 raft_consensus.cc:686] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 1 LEADER]: Becoming Leader. State: Replica: 5f3b749d236b4f36838b6dcd815c511e, State: Running, Role: LEADER
I20260929 14:50:05.498232 13637 consensus_queue.cc:237] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } }
I20260929 14:50:05.502060 13135 catalog_manager.cc:5495] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3b749d236b4f36838b6dcd815c511e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5f3b749d236b4f36838b6dcd815c511e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3b749d236b4f36838b6dcd815c511e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34739 } health_report { overall_health: HEALTHY } } }
I20260929 14:50:05.529013 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.523957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.98.2:34739)
0929 14:50:05.524112 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:05.524116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:05.524130 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:50:05.524243 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:05.524246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:05.524347 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:05.524458 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:05.524884 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.525681 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.527046 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:05.527366 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:05.527471 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:05.527887 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:05.528496 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:05.528655 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:05.528776 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 14:50:05.528843 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:50:05.538419 13544 raft_consensus.cc:1261] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 1 FOLLOWER]: Refusing update from remote peer 5f3b749d236b4f36838b6dcd815c511e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:05.538576 13285 raft_consensus.cc:1261] T b0e11bbef08f417cbd87b8afee52b0bc P 2ae1ad91dec74820a8a6ad84b95050ed [term 1 FOLLOWER]: Refusing update from remote peer 5f3b749d236b4f36838b6dcd815c511e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:05.539619 13645 consensus_queue.cc:1035] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af03e54bd234d0894692030c6555b06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:05.539851 13637 consensus_queue.cc:1035] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae1ad91dec74820a8a6ad84b95050ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:05.553978 13648 mvcc.cc:205] Tried to move back new op lower bound from 7334680189074370560 to 7334680188923518976. Current Snapshot: MvccSnapshot[applied={T|T < 7334680189074370560}]
I20260929 14:50:05.558090 13650 mvcc.cc:205] Tried to move back new op lower bound from 7334680189074370560 to 7334680188923518976. Current Snapshot: MvccSnapshot[applied={T|T < 7334680189074370560}]
I20260929 14:50:05.560580 13649 mvcc.cc:205] Tried to move back new op lower bound from 7334680189074370560 to 7334680188923518976. Current Snapshot: MvccSnapshot[applied={T|T < 7334680189074370560}]
I20260929 14:50:05.568092 13134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:05.568464 13134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:05.571913 13134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea990a812bc04dae957f441e288a4ff2: Sending DeleteTablet for 3 replicas of tablet b0e11bbef08f417cbd87b8afee52b0bc
I20260929 14:50:05.573060 13536 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55874
I20260929 14:50:05.573483 13266 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45022
I20260929 14:50:05.574781 13402 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e11bbef08f417cbd87b8afee52b0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36836
I20260929 14:50:05.576488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13173
I20260929 14:50:05.581883 13659 tablet_replica.cc:330] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: stopping tablet replica
I20260929 14:50:05.582420 13659 raft_consensus.cc:2229] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:50:05.589428 13659 raft_consensus.cc:2258] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:05.590058 13657 tablet_replica.cc:330] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: stopping tablet replica
I20260929 14:50:05.592042 13659 ts_tablet_manager.cc:1905] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:05.595448 13659 ts_tablet_manager.cc:1918] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:05.596030 13657 raft_consensus.cc:2229] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:05.596612 13657 raft_consensus.cc:2258] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:05.597397 13659 log.cc:1196] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/b0e11bbef08f417cbd87b8afee52b0bc
W20260929 14:50:05.597749 13120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41873 (error 108)
I20260929 14:50:05.599133 13659 ts_tablet_manager.cc:1939] T b0e11bbef08f417cbd87b8afee52b0bc P 5f3b749d236b4f36838b6dcd815c511e: Deleting consensus metadata
I20260929 14:50:05.599447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13337
I20260929 14:50:05.600145 13657 ts_tablet_manager.cc:1905] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:05.604961 13657 ts_tablet_manager.cc:1918] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:05.605108 13657 log.cc:1196] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/b0e11bbef08f417cbd87b8afee52b0bc
I20260929 14:50:05.605499 13657 ts_tablet_manager.cc:1939] T b0e11bbef08f417cbd87b8afee52b0bc P 1af03e54bd234d0894692030c6555b06: Deleting consensus metadata
I20260929 14:50:05.605628 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:05.605114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41873 (local address 127.0.0.1:45028)
0929 14:50:05.605422 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.605506 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260929 14:50:05.605930 13120 catalog_manager.cc:4584] TS 2ae1ad91dec74820a8a6ad84b95050ed (127.0.98.1:41873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0e11bbef08f417cbd87b8afee52b0bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41873: connect: Connection refused (error 111)
I20260929 14:50:05.607272 13119 catalog_manager.cc:4841] TS 1af03e54bd234d0894692030c6555b06 (127.0.98.3:38631): tablet b0e11bbef08f417cbd87b8afee52b0bc (table test-table [id=00d8c43c72d447aa9bebf693c9c237ba]) successfully deleted
W20260929 14:50:05.613536 13629 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:50:05.612866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34739 (local address 127.0.0.1:36848)
0929 14:50:05.613047 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:05.613065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:05.613170 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:05.613408 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34739: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 14:50:05.614395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13471
I20260929 14:50:05.623863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13102
I20260929 14:50:05.631253 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:50:05.977386 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:50:05.977545 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:50:06.322582 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:50:06.322 UTC [12169] LOG: received smart shutdown request
2026-09-29 14:50:06.326 UTC [12169] LOG: background worker "logical replication launcher" (PID 12178) exited with exit code 1
2026-09-29 14:50:06.327 UTC [12173] LOG: shutting down
2026-09-29 14:50:06.554 UTC [12169] LOG: database system is shut down
2026-09-29T14:50:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53099 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:50:06 dist-test-slave-5hd1 krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:50:06 dist-test-slave-5hd1 krb5kdc[13669](info): set up 2 sockets
Sep 29 14:50:06 dist-test-slave-5hd1 krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260929 14:50:07.728669 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:50:07 dist-test-slave-5hd1 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:50:07.867699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41017
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:07.992842 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:07.993167 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:07.993211 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:07.997288 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:07.997363 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:07.997382 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:07.997400 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:07.997417 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:08.003393 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:08.004776 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:08.005991 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:08.016216 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:08.016268 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:08.016542 13682 server_base.cc:1034] running on GCE node
W20260929 14:50:08.016439 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:08.017345 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:08.018618 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:08.019798 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790693408019777 us; error 42 us; skew 500 ppm
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:08.022907 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:08.024060 13682 webserver.cc:460] Webserver started at http://127.0.98.62:38771/ using document root <none> and password file <none>
I20260929 14:50:08.024784 13682 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:08.024857 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:08.025122 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:08.027873 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "3f11c7169445432086974801a6141974"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "ab97a5400ccf0611d26c577c94006fcf"
server_key_iv: "29cb6fa61d717530aea63fe6bf5f4d90"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.028692 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f11c7169445432086974801a6141974"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "ab97a5400ccf0611d26c577c94006fcf"
server_key_iv: "29cb6fa61d717530aea63fe6bf5f4d90"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.033627 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:50:08.036794 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:08.038611 13682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:08.038813 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "3f11c7169445432086974801a6141974"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "ab97a5400ccf0611d26c577c94006fcf"
server_key_iv: "29cb6fa61d717530aea63fe6bf5f4d90"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.038970 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:08.064576 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:08.068329 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:08.068581 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:08.078086 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41017
I20260929 14:50:08.078126 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41017 every 8 connection(s)
I20260929 14:50:08.079290 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:08.082751 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:08.085665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13682
I20260929 14:50:08.085850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:50:08.086352 392 external_mini_cluster.cc:1428] Setting key 81bd8f6a26e52c3bf8467d56be2a45e5
I20260929 14:50:08.087049 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:08.109193 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.089697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.0.98.62:41017)
0929 14:50:08.090355 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.090372 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.090440 (+ 68us) server_negotiation.cc:407] Connection header received
0929 14:50:08.092010 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.092020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.092556 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.092791 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.094203 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.095179 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.096988 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.097409 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.097548 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.102509 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.102519 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.102539 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.102583 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.104870 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.105468 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.105477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.105481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.105634 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.105987 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.105994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.105999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.106611 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.106777 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.107340 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.107720 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":506,"thread_start_us":330,"threads_started":1}
I20260929 14:50:08.109189 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.089696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41017 (local address 127.0.0.1:54902)
0929 14:50:08.090281 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.090342 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.091544 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.092842 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.092848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.093411 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.093925 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.093954 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.095356 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.095360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.096793 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.096808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.097620 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.097627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.097802 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.098523 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.098542 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.102258 (+ 3716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.104999 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.105008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.105019 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.105328 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.105744 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.105747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.105748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.105873 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.106769 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.106777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.106952 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.107224 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.107944 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":166,"thread_start_us":171,"threads_started":1}
I20260929 14:50:08.112381 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:50:08.113375 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.113024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59774)
0929 14:50:08.113198 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.113288 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 14:50:08.113683 13698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
I20260929 14:50:08.114691 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.114477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59778)
0929 14:50:08.114584 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.114636 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 14:50:08.114985 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39481: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:50:08.121014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39481
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:08.164325 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } attempt: 1
I20260929 14:50:08.165612 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.165212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59784)
0929 14:50:08.165430 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.165524 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 14:50:08.166505 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.166285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59796)
0929 14:50:08.166389 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.166451 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:50:08.166637 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39481: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
I20260929 14:50:08.234855 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } attempt: 2
I20260929 14:50:08.235970 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.235583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59800)
0929 14:50:08.235811 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.235891 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 14:50:08.236971 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.236737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59808)
0929 14:50:08.236862 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.236917 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 14:50:08.237102 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39481: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
W20260929 14:50:08.259598 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:08.259903 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:08.259948 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:08.264159 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:08.264264 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:08.264285 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:08.264303 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:08.264350 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:08.270191 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:08.271664 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:08.272989 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:08.283452 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:08.283538 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:08.283452 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:08.283818 13757 server_base.cc:1034] running on GCE node
I20260929 14:50:08.284674 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:08.285952 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:08.287143 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790693408287126 us; error 40 us; skew 500 ppm
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:08.290349 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:50:08.291347 13757 webserver.cc:460] Webserver started at http://127.0.98.61:35579/ using document root <none> and password file <none>
I20260929 14:50:08.291985 13757 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:08.292080 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:08.292371 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:08.294489 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data/instance:
uuid: "3e2c4a3e9e964188a1f5510cd3239232"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "1e7da03656a2ec45339063c253894bf3"
server_key_iv: "82959e079dfa0e08c6af23a616fbce42"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.295101 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal/instance:
uuid: "3e2c4a3e9e964188a1f5510cd3239232"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "1e7da03656a2ec45339063c253894bf3"
server_key_iv: "82959e079dfa0e08c6af23a616fbce42"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.299329 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:50:08.302266 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:08.303812 13757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:08.303972 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "3e2c4a3e9e964188a1f5510cd3239232"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "1e7da03656a2ec45339063c253894bf3"
server_key_iv: "82959e079dfa0e08c6af23a616fbce42"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.304087 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:08.319864 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:08.322786 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:08.322997 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:08.328272 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } attempt: 3
I20260929 14:50:08.329272 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.328860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59822)
0929 14:50:08.329061 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.329151 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 14:50:08.330224 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.329968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59834)
0929 14:50:08.330089 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.330151 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 14:50:08.330389 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39481: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39481: connect: Connection refused (error 111)
I20260929 14:50:08.330955 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39481
I20260929 14:50:08.330972 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39481 every 8 connection(s)
I20260929 14:50:08.331974 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:50:08.335395 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:08.339607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13757
I20260929 14:50:08.339766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-1/wal/instance
I20260929 14:50:08.340121 392 external_mini_cluster.cc:1428] Setting key 34578a1c7c88c66f19ba49e879a361d9
I20260929 14:50:08.339974 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:08.357710 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.342643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.0.98.62:41017)
0929 14:50:08.342819 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.342824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.344150 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 14:50:08.345316 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.345319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.345392 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.345499 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.347016 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.347633 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.348931 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.349254 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.349366 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.352755 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.352758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.352761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.352786 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.354729 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.355478 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.355489 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.355491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.355641 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.356142 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.356146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.356147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.356458 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.356565 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.357406 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.357551 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:50:08.359264 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.343267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41017 (local address 127.0.0.1:54906)
0929 14:50:08.343983 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.344032 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.345059 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.345689 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.345702 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.346097 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.346667 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.346681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.347783 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.347787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.348806 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.348817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.349426 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.349430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.349607 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.350368 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.350386 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.352557 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.354851 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.354861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.354881 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.355282 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.355767 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.355773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.355777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.355979 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.356595 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.356605 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.357050 (+ 445us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.357318 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.357774 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":167,"threads_started":1}
I20260929 14:50:08.361333 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:50:08.372357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37813
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:08.372893 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.362084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59838)
0929 14:50:08.362349 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.362369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.362484 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.363393 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.363395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.363606 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.363734 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.363740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.364948 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.364955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.365569 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.365577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.366337 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.366340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.366442 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.367041 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.367054 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.368691 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.370966 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.370969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.370971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.371247 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.371647 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.371649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.371651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.371732 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.372350 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.372353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.372451 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.372569 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.372736 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 14:50:08.373418 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.362281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.98.61:39481)
0929 14:50:08.362916 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.362924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.362992 (+ 68us) server_negotiation.cc:407] Connection header received
0929 14:50:08.363056 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.363062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.363249 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.363374 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.363859 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.364834 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.365685 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.366176 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.366324 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.368875 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.368885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.368898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.368929 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.370839 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.371439 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.371446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.371448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.371541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.371828 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.371831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.371832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.372230 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.372385 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.372811 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.373065 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":228,"threads_started":1}
I20260929 14:50:08.376016 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:50:08.376927 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.376614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37812)
0929 14:50:08.376770 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.376861 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 14:50:08.377283 13774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
I20260929 14:50:08.378392 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.378166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37828)
0929 14:50:08.378291 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.378340 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 14:50:08.378731 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
I20260929 14:50:08.428045 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } attempt: 1
I20260929 14:50:08.429034 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.428748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37832)
0929 14:50:08.428896 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.428962 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 14:50:08.429745 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.429545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37844)
0929 14:50:08.429648 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.429692 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 14:50:08.429893 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
I20260929 14:50:08.473598 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } attempt: 4
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:50:08.483667 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.474421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59850)
0929 14:50:08.474696 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.474717 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.474860 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.475158 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.475161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.475362 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.475920 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.475927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.476774 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.476780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.477317 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.477324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.478244 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.478248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.478354 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.479015 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.479029 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.480371 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.482042 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.482048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.482051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.482311 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.482617 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.482620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.482622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.482701 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.483111 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.483114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.483238 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.483347 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.483483 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 14:50:08.483667 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.474597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.0.98.61:39481)
0929 14:50:08.474765 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.474772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.474789 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:50:08.474980 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.474983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.475037 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.475131 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.476049 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.476663 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.477436 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.478113 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.478228 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.480516 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.480519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.480521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.480541 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.481930 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.482431 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.482433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.482434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.482502 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.482792 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.482797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.482799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.482977 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.483080 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.483395 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.483502 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 14:50:08.484877 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:50:08.485579 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.485356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37860)
0929 14:50:08.485462 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.485511 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 14:50:08.486164 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.485962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37872)
0929 14:50:08.486056 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.486120 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 14:50:08.486306 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
I20260929 14:50:08.498096 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } attempt: 2
I20260929 14:50:08.499068 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.498762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37886)
0929 14:50:08.498925 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.498990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 14:50:08.499770 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.499580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37892)
0929 14:50:08.499675 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.499726 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 14:50:08.499902 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
W20260929 14:50:08.503763 13831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:08.504148 13831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:08.504268 13831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:08.509017 13831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:08.509107 13831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:08.509145 13831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:08.509203 13831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:08.509249 13831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:08.515494 13831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41017,127.0.98.61:39481,127.0.98.60:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:08.517112 13831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:08.518429 13831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:08.528506 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:08.528564 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:08.528812 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:08.529099 13831 server_base.cc:1034] running on GCE node
I20260929 14:50:08.529891 13831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:08.531207 13831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:08.532411 13831 hybrid_clock.cc:648] HybridClock initialized: now 1790693408532392 us; error 40 us; skew 500 ppm
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:08.535389 13831 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:50:08.536420 13831 webserver.cc:460] Webserver started at http://127.0.98.60:40747/ using document root <none> and password file <none>
I20260929 14:50:08.536988 13831 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:08.537055 13831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:08.537230 13831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:08.539309 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data/instance:
uuid: "ddbdd48999bb4d3089b8510396d0369b"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "6f199b97724c3c49aa32c03d76dd3655"
server_key_iv: "f5d524c5fdda7336d6746e103af28e83"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.539846 13831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal/instance:
uuid: "ddbdd48999bb4d3089b8510396d0369b"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "6f199b97724c3c49aa32c03d76dd3655"
server_key_iv: "f5d524c5fdda7336d6746e103af28e83"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.544291 13831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:50:08.545486 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } attempt: 1
I20260929 14:50:08.546408 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.546061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37906)
0929 14:50:08.546263 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.546331 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 14:50:08.547165 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.546952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37920)
0929 14:50:08.547051 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.547102 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:50:08.547390 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37813: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37813: connect: Connection refused (error 111)
I20260929 14:50:08.547417 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:08.549041 13831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:08.549229 13831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "ddbdd48999bb4d3089b8510396d0369b"
format_stamp: "Formatted at 2026-09-29 14:50:08 on dist-test-slave-5hd1"
server_key: "6f199b97724c3c49aa32c03d76dd3655"
server_key_iv: "f5d524c5fdda7336d6746e103af28e83"
server_key_version: "encryptionkey@0"
I20260929 14:50:08.549392 13831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:08.575713 13831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:08.579968 13831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:08.580255 13831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:08.589053 13831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37813
I20260929 14:50:08.589104 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37813 every 8 connection(s)
I20260929 14:50:08.590234 13831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:50:08.591101 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } attempt: 3
I20260929 14:50:08.591840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13831
I20260929 14:50:08.591984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790693238534145-392-0/minicluster-data/master-2/wal/instance
I20260929 14:50:08.592360 392 external_mini_cluster.cc:1428] Setting key 4533b1bd586616638018ea175cf71c7f
I20260929 14:50:08.594266 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:08.599239 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:08.612044 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.591795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37936)
0929 14:50:08.592068 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.592082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.592226 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.598429 (+ 6203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.598433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.598711 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.598856 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.598864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.600838 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.600842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.601670 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.601680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.603035 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.603042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.603191 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.603872 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.603894 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.605869 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.609863 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.609868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.609872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.610157 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.610574 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.610576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.610577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.610654 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.611364 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.611370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.611509 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.611624 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.611802 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 14:50:08.613070 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.601398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.98.62:41017)
0929 14:50:08.602238 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.602242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.602254 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:50:08.602293 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.602295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.602334 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.602436 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.603283 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.604134 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.605221 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.605470 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.605610 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.609038 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.609042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.609045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.609076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.610682 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.611233 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.611238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.611240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.611344 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.611621 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.611626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.611628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.611863 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.611971 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.612749 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.612905 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753}
I20260929 14:50:08.613957 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.593547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.0.98.60:37813)
0929 14:50:08.596881 (+ 3334us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.596887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.596897 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:50:08.597518 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.597537 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.598045 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.598490 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.599859 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.600602 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.601845 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.602867 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.603153 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.606806 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.606812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.606817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.606854 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.610437 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.611020 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.611024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.611026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.611110 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.611418 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.611421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.611422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.611645 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.611821 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.612483 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.613145 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":622,"server-negotiator.queue_time_us":3279,"spinlock_wait_cycles":27904,"thread_start_us":70,"threads_started":1}
I20260929 14:50:08.613970 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.601344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41017 (local address 127.0.0.1:54918)
0929 14:50:08.601822 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.601878 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.602003 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.602492 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.602498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.602888 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.603128 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.603138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.604281 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.604284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.605067 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.605077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.605676 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.605681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.605845 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.606455 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.606485 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.608841 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.610808 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.610819 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.610828 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.611116 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.611437 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.611442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.611444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.611526 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.611978 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.611986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.612474 (+ 488us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.612644 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.612849 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":3,"thread_start_us":212,"threads_started":1}
I20260929 14:50:08.613970 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.593006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.98.60:37813)
0929 14:50:08.596461 (+ 3455us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.596483 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.596525 (+ 42us) server_negotiation.cc:407] Connection header received
0929 14:50:08.597499 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.597533 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.598052 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.598490 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.599073 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.600610 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.601839 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.602867 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.603093 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.606023 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.606030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.606053 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.606107 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.609717 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.610289 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.610295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.610301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.610437 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.610752 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.610758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.610760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.611222 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.611356 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.612419 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.612799 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":3281,"spinlock_wait_cycles":13696,"thread_start_us":183,"threads_started":1}
I20260929 14:50:08.614598 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } attempt: 2
I20260929 14:50:08.615484 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:50:08.619626 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: Bootstrap starting.
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:08.624543 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:08.625639 13825 log.cc:824] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:08.626125 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.615028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37958)
0929 14:50:08.615194 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.615213 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.615305 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.615707 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.615710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.615935 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.616052 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.616059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.617149 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.617152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.617878 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.617888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.618520 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.618523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.618602 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.619137 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.619154 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.620921 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.623251 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.623257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.623262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.623555 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.623886 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.623892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.623894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.624302 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.625582 (+ 1280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.625588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.625694 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.625783 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.625930 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 14:50:08.626762 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.615283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.0.98.60:37813)
0929 14:50:08.615488 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.615491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.615501 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:50:08.615526 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.615528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.615561 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.616165 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.616282 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.617015 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.618033 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.618388 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.618833 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.621038 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.621042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.621045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.621071 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.623122 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.623680 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.623685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.623687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.623777 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.624910 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.624915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.624919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.625446 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.626346 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.626422 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.626580 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":966,"server-negotiator.queue_time_us":148}
I20260929 14:50:08.627875 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.616952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.0.98.62:41017)
0929 14:50:08.617053 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.617058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.617208 (+ 150us) server_negotiation.cc:407] Connection header received
0929 14:50:08.617379 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.617382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.617428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.617477 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.618325 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.619025 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.620111 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.620411 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.620524 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.623683 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.623686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.623690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.623714 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.625562 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626047 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.626051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.626053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.626130 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626557 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.626560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.626562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.626756 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.627017 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.627618 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.627737 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:50:08.627939 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.616500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.98.61:39481)
0929 14:50:08.616641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.616646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.616804 (+ 158us) server_negotiation.cc:407] Connection header received
0929 14:50:08.616986 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.616990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.617053 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.617112 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.617733 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.618545 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.620033 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.620427 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.620545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.624068 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.624071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.624074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.624100 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.625887 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626482 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.626486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.626488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.626574 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626900 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.626903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.626905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.627188 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.627275 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.627642 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.627768 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:50:08.628436 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.616001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39481 (local address 127.0.0.1:59852)
0929 14:50:08.616736 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.616754 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.616872 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.617274 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.617277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.617479 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.617603 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.617611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.618677 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.618680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.619897 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.619908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.620675 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.620678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.620788 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.621348 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.621361 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.623944 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.626017 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.626023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.626368 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.626692 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.626695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.626697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.626789 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.627348 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.627351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.627441 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.628070 (+ 629us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.628280 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662}
I20260929 14:50:08.629256 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:50:08.631831 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: Bootstrap starting.
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:50:08.635298 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:08.636322 13750 log.cc:824] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:08.637085 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: No bootstrap required, opened a new log
I20260929 14:50:08.638336 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.629806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37813 (local address 127.0.0.1:37972)
0929 14:50:08.629980 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:08.629995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:08.630078 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:08.630310 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:08.630313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:08.630503 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:08.630616 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.630626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.631604 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.631607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.632272 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:08.632281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.632899 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.632903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:08.632975 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.633412 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:08.633430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:08.634828 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:08.636857 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.636859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.636861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.637095 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.637370 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:08.637372 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:08.637374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:08.637442 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:08.637858 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:08.637861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:08.637934 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 14:50:08.638024 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 14:50:08.638178 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 14:50:08.638774 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.629878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.0.98.60:37813)
0929 14:50:08.629994 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.630000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.630045 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:50:08.630165 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.630167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.630200 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.630268 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.630748 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.631486 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.632390 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.632692 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.632797 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.634969 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.634973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.634977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.635001 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.636739 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.637208 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.637211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.637213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.637285 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.637537 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.637540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.637542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.637736 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.637826 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.638541 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.638644 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:50:08.641306 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: No bootstrap required, opened a new log
I20260929 14:50:08.642432 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: Bootstrap starting.
I20260929 14:50:08.642271 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.642634 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:08.642685 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2c4a3e9e964188a1f5510cd3239232, State: Initialized, Role: FOLLOWER
I20260929 14:50:08.643347 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
Sep 29 14:50:08 dist-test-slave-5hd1 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:50:08.645428 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:08.645867 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.646057 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:08.646265 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:08.646392 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f11c7169445432086974801a6141974, State: Initialized, Role: FOLLOWER
I20260929 14:50:08.647004 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.647099 13900 log.cc:824] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:08.648100 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.637186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.98.61:39481)
0929 14:50:08.637330 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.637334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.637415 (+ 81us) server_negotiation.cc:407] Connection header received
0929 14:50:08.637626 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.637630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.637687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.637776 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.638715 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.639342 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.640599 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.640957 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.641072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.644058 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.644062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.644065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.644091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.645752 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.646255 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.646261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.646264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.646352 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.646692 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.646695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.646697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.646929 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.647106 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.647781 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.647903 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:50:08.650002 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: No bootstrap required, opened a new log
I20260929 14:50:08.650290 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:08.652323 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:50:08.649569 13757 master_runner.cc:186] Error getting master registration for 127.0.98.62:41017: OK, instance_id { permanent_uuid: "3f11c7169445432086974801a6141974" instance_seqno: 1790693408075286 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:50:08.653393 13757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:50:08.654939 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.655337 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:08.655422 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbdd48999bb4d3089b8510396d0369b, State: Initialized, Role: FOLLOWER
I20260929 14:50:08.656049 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.657680 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.657907 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.658123 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:08.658354 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.658511 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.661362 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.661594 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: This master's current role is: FOLLOWER
W20260929 14:50:08.662978 13921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:50:08.663066 13921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:50:08.664862 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.654905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.61:39481)
0929 14:50:08.655052 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.655057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.655424 (+ 367us) server_negotiation.cc:407] Connection header received
0929 14:50:08.655626 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.655630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.655684 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.655753 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.657014 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.657703 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.658795 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.659125 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.659198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.661096 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.661101 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.661104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.661131 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.662827 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.663402 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.663412 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.663414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.663507 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.663825 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.663829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.663830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.664071 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.664209 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.664575 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.664705 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:50:08.667028 13682 master_runner.cc:263] No leader master found from master 3f11c7169445432086974801a6141974
I20260929 14:50:08.667104 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:50:08.667858 13831 master_runner.cc:263] No leader master found from master ddbdd48999bb4d3089b8510396d0369b
I20260929 14:50:08.667943 13831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 14:50:08.669661 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:50:08.669831 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 14:50:08.674726 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:50:08.674958 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:50:08.675937 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.654876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.0.98.62:41017)
0929 14:50:08.654997 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.655003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.662256 (+ 7253us) server_negotiation.cc:407] Connection header received
0929 14:50:08.662482 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.662486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.662541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.662598 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.663768 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.664602 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.669296 (+ 4694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.669670 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.669760 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.671940 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.671946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.671950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.671984 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.673747 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.674283 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.674286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.674288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.674371 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.674698 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.674704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.674709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.674959 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.675084 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.675553 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.675741 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:50:08.676437 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:08.657019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.0.98.60:37813)
0929 14:50:08.657132 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:08.657137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:08.667419 (+ 10282us) server_negotiation.cc:407] Connection header received
0929 14:50:08.667582 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:08.667586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:08.667630 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:08.667683 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:08.668598 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.669305 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.670302 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:08.670665 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:08.670767 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:08.672686 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:08.672691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:08.672696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:08.672728 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:08.674637 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.675080 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.675083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.675085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.675167 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:08.675443 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:08.675446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:08.675448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:08.675682 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:08.675812 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:08.676120 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 14:50:08.676264 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:50:08.727497 13908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:08.727707 13908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.729156 13908 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f11c7169445432086974801a6141974 (127.0.98.62:41017), ddbdd48999bb4d3089b8510396d0369b (127.0.98.60:37813)
I20260929 14:50:08.729871 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e2c4a3e9e964188a1f5510cd3239232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f11c7169445432086974801a6141974" is_pre_election: true
I20260929 14:50:08.730221 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2c4a3e9e964188a1f5510cd3239232 in term 0.
I20260929 14:50:08.730811 13775 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2c4a3e9e964188a1f5510cd3239232, 3f11c7169445432086974801a6141974; no voters:
I20260929 14:50:08.731649 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e2c4a3e9e964188a1f5510cd3239232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdd48999bb4d3089b8510396d0369b" is_pre_election: true
I20260929 14:50:08.731976 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2c4a3e9e964188a1f5510cd3239232 in term 0.
I20260929 14:50:08.732793 13908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:08.732880 13908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:08.732926 13908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:08.734472 13908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.735203 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e2c4a3e9e964188a1f5510cd3239232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f11c7169445432086974801a6141974"
I20260929 14:50:08.735369 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:08.736897 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2c4a3e9e964188a1f5510cd3239232 in term 1.
I20260929 14:50:08.737306 13775 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2c4a3e9e964188a1f5510cd3239232, 3f11c7169445432086974801a6141974; no voters:
I20260929 14:50:08.740285 13908 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [CANDIDATE]: Term 1 election: Requested vote from peers 3f11c7169445432086974801a6141974 (127.0.98.62:41017), ddbdd48999bb4d3089b8510396d0369b (127.0.98.60:37813)
I20260929 14:50:08.740679 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e2c4a3e9e964188a1f5510cd3239232" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbdd48999bb4d3089b8510396d0369b"
I20260929 14:50:08.740813 13874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:08.742372 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2c4a3e9e964188a1f5510cd3239232 in term 1.
I20260929 14:50:08.742946 13908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:08.743139 13908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2c4a3e9e964188a1f5510cd3239232, State: Running, Role: LEADER
I20260929 14:50:08.743571 13908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } }
I20260929 14:50:08.745777 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2c4a3e9e964188a1f5510cd3239232. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.745977 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:08.746496 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:08.747663 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:08.758167 13874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [term 1 FOLLOWER]: Refusing update from remote peer 3e2c4a3e9e964188a1f5510cd3239232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:08.758998 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:08.763450 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2c4a3e9e964188a1f5510cd3239232. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.763579 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.767184 13724 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [term 1 FOLLOWER]: Refusing update from remote peer 3e2c4a3e9e964188a1f5510cd3239232: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:08.767802 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:08.772115 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2c4a3e9e964188a1f5510cd3239232. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.772261 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.778250 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.778404 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:08.779757 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.779873 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.782717 13958 mvcc.cc:205] Tried to move back new op lower bound from 7334680202264952832 to 7334680202218098688. Current Snapshot: MvccSnapshot[applied={T|T < 7334680202264952832}]
I20260929 14:50:08.784418 13908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.784534 13908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:08.785549 13939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e2c4a3e9e964188a1f5510cd3239232" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f11c7169445432086974801a6141974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41017 } } peers { permanent_uuid: "3e2c4a3e9e964188a1f5510cd3239232" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39481 } } peers { permanent_uuid: "ddbdd48999bb4d3089b8510396d0369b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37813 } } }
I20260929 14:50:08.785670 13939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:50:08.787998 13959 mvcc.cc:205] Tried to move back new op lower bound from 7334680202264952832 to 7334680202218098688. Current Snapshot: MvccSnapshot[applied={T|T < 7334680202264952832}]
I20260929 14:50:08.788213 13961 mvcc.cc:205] Tried to move back new op lower bound from 7334680202264952832 to 7334680202218098688. Current Snapshot: MvccSnapshot[applied={T|T < 7334680202264952832}]
I20260929 14:50:08.791858 13957 catalog_manager.cc:1347] Generated new cluster ID: e2fa6515659144d990e962202dc9ba6a
I20260929 14:50:08.791921 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:08.823498 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:50:08.825155 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:08.845364 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2c4a3e9e964188a1f5510cd3239232: Generated new TSK 0
I20260929 14:50:08.849239 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:50:09.671736 13938 catalog_manager.cc:1259] Loaded cluster ID: e2fa6515659144d990e962202dc9ba6a
I20260929 14:50:09.671901 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: loading cluster ID for follower catalog manager: success
I20260929 14:50:09.674130 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: acquiring CA information for follower catalog manager: success
I20260929 14:50:09.675729 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f11c7169445432086974801a6141974: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:50:09.676525 13951 catalog_manager.cc:1259] Loaded cluster ID: e2fa6515659144d990e962202dc9ba6a
I20260929 14:50:09.676620 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: loading cluster ID for follower catalog manager: success
I20260929 14:50:09.678318 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: acquiring CA information for follower catalog manager: success
I20260929 14:50:09.679502 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddbdd48999bb4d3089b8510396d0369b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:50:09.928675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13682
I20260929 14:50:09.937361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13757
I20260929 14:50:09.944924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13831
2026-09-29T14:50:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3274 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:50:10 dist-test-slave-5hd1 krb5kdc[13973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:50:10 dist-test-slave-5hd1 krb5kdc[13973](info): set up 2 sockets
Sep 29 14:50:10 dist-test-slave-5hd1 krb5kdc[13973](info): commencing operation
krb5kdc: starting...
W20260929 14:50:10.996594 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:50:11.132051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:11.246469 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:11.246738 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:11.246776 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:11.250869 13986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:11.250931 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:11.250952 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:11.250967 13986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:11.250982 13986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:11.256196 13986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:11.257236 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:11.258260 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:11.267190 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:11.267218 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:11.267190 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:11.267524 13986 server_base.cc:1034] running on GCE node
I20260929 14:50:11.268513 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:11.269917 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:11.271124 13986 hybrid_clock.cc:648] HybridClock initialized: now 1790693411271100 us; error 47 us; skew 500 ppm
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:11.274448 13986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:11.275588 13986 webserver.cc:460] Webserver started at http://127.0.98.62:41999/ using document root <none> and password file <none>
I20260929 14:50:11.276320 13986 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:11.276402 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:11.276618 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:11.279218 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "4fdfe31748e944009662ba72a748245f"
format_stamp: "Formatted at 2026-09-29 14:50:11 on dist-test-slave-5hd1"
server_key: "f654fe23382d2e7f6db21445836329d8"
server_key_iv: "90dc03b821b72df608d3657c3c0bb596"
server_key_version: "encryptionkey@0"
I20260929 14:50:11.279850 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "4fdfe31748e944009662ba72a748245f"
format_stamp: "Formatted at 2026-09-29 14:50:11 on dist-test-slave-5hd1"
server_key: "f654fe23382d2e7f6db21445836329d8"
server_key_iv: "90dc03b821b72df608d3657c3c0bb596"
server_key_version: "encryptionkey@0"
I20260929 14:50:11.284830 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 14:50:11.287768 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:11.289358 13986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:11.289489 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "4fdfe31748e944009662ba72a748245f"
format_stamp: "Formatted at 2026-09-29 14:50:11 on dist-test-slave-5hd1"
server_key: "f654fe23382d2e7f6db21445836329d8"
server_key_iv: "90dc03b821b72df608d3657c3c0bb596"
server_key_version: "encryptionkey@0"
I20260929 14:50:11.289600 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:11.315173 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:11.318746 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:11.318928 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:11.328012 13986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46035
I20260929 14:50:11.328044 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46035 every 8 connection(s)
I20260929 14:50:11.329164 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:11.332296 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:11.337826 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f: Bootstrap starting.
I20260929 14:50:11.339054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 13986
I20260929 14:50:11.339181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:50:11.339516 392 external_mini_cluster.cc:1428] Setting key dc7ed4091207045547983e6fa94903f2
I20260929 14:50:11.340730 14054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:11.341574 14054 log.cc:824] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:11.343716 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f: No bootstrap required, opened a new log
I20260929 14:50:11.347357 14054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } }
I20260929 14:50:11.347630 14054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:11.347692 14054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdfe31748e944009662ba72a748245f, State: Initialized, Role: FOLLOWER
I20260929 14:50:11.348234 14054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } }
I20260929 14:50:11.348380 14054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:11.348507 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:11.348628 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:11.349565 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } }
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[13973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:11.349910 14054 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fdfe31748e944009662ba72a748245f; no voters:
I20260929 14:50:11.350272 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:50:11.350349 14059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:11.350565 14059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [term 1 LEADER]: Becoming Leader. State: Replica: 4fdfe31748e944009662ba72a748245f, State: Running, Role: LEADER
I20260929 14:50:11.351174 14059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } }
I20260929 14:50:11.351600 14054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:11.352866 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fdfe31748e944009662ba72a748245f. Latest consensus state: current_term: 1 leader_uuid: "4fdfe31748e944009662ba72a748245f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } } }
I20260929 14:50:11.353015 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:11.353331 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fdfe31748e944009662ba72a748245f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fdfe31748e944009662ba72a748245f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46035 } } }
I20260929 14:50:11.353471 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:11.353755 14068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:11.356942 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:11.340876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.0.98.62:46035)
0929 14:50:11.341432 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:11.341454 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:11.341502 (+ 48us) server_negotiation.cc:407] Connection header received
0929 14:50:11.342631 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:11.342655 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:11.343088 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:11.343502 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:11.344455 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:11.345605 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:11.346501 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:11.347028 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:11.347214 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:11.350421 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:11.350432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:11.350456 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:11.350500 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:11.352850 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:11.353507 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:11.353513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:11.353520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:11.353664 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:11.354086 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:11.354092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:11.354095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:11.354656 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:11.355012 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:11.355493 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 14:50:11.355981 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":167,"threads_started":1}
I20260929 14:50:11.357677 14068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:11.360340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:50:11.364658 14068 catalog_manager.cc:1347] Generated new cluster ID: 6fe2b0881b404f94816c5dbe10d99643
I20260929 14:50:11.364730 14068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:11.383051 14068 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:50:11.384472 14068 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:11.396569 14068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fdfe31748e944009662ba72a748245f: Generated new TSK 0
I20260929 14:50:11.397545 14068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:50:11.417249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 13986
2026-09-29T14:50:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1469 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[14081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[14081](info): set up 2 sockets
Sep 29 14:50:11 dist-test-slave-5hd1 krb5kdc[14081](info): commencing operation
krb5kdc: starting...
W20260929 14:50:12.448428 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:50:12 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:50:12.588392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:12.715268 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:12.715633 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:12.715718 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:12.720048 14094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:12.720160 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:12.720207 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:12.720240 14094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:12.720276 14094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:12.726061 14094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:12.727473 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:12.728744 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:12.738756 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:12.738763 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:12.738878 14094 server_base.cc:1034] running on GCE node
W20260929 14:50:12.738752 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:12.739737 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:12.741114 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:12.742347 14094 hybrid_clock.cc:648] HybridClock initialized: now 1790693412742311 us; error 50 us; skew 500 ppm
Sep 29 14:50:12 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:12.745723 14094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:12.746855 14094 webserver.cc:460] Webserver started at http://127.0.98.62:34501/ using document root <none> and password file <none>
I20260929 14:50:12.747532 14094 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:12.747630 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:12.747900 14094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:12.750280 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c"
format_stamp: "Formatted at 2026-09-29 14:50:12 on dist-test-slave-5hd1"
server_key: "fc59d17d34b27ebd14614fd59a6ada55"
server_key_iv: "45d01d8b15c85a8edd282e49b9715f6a"
server_key_version: "encryptionkey@0"
I20260929 14:50:12.750903 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c"
format_stamp: "Formatted at 2026-09-29 14:50:12 on dist-test-slave-5hd1"
server_key: "fc59d17d34b27ebd14614fd59a6ada55"
server_key_iv: "45d01d8b15c85a8edd282e49b9715f6a"
server_key_version: "encryptionkey@0"
I20260929 14:50:12.755162 14094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:50:12.757944 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:12.759464 14094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:50:12.759615 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c"
format_stamp: "Formatted at 2026-09-29 14:50:12 on dist-test-slave-5hd1"
server_key: "fc59d17d34b27ebd14614fd59a6ada55"
server_key_iv: "45d01d8b15c85a8edd282e49b9715f6a"
server_key_version: "encryptionkey@0"
I20260929 14:50:12.759733 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:12.791108 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:12.795029 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:12.795257 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:12.803896 14094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37801
I20260929 14:50:12.804329 14161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37801 every 8 connection(s)
I20260929 14:50:12.805080 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:12.805970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14094
I20260929 14:50:12.806128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:50:12.806478 392 external_mini_cluster.cc:1428] Setting key d673fb571e9854973e4b65ffb040f07f
I20260929 14:50:12.808495 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:12.815245 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap starting.
Sep 29 14:50:12 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:12.817677 14162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:12.818593 14162 log.cc:824] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:12.821197 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: No bootstrap required, opened a new log
I20260929 14:50:12.823293 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:12.808270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.98.62:37801)
0929 14:50:12.808783 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:12.808804 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:12.808853 (+ 49us) server_negotiation.cc:407] Connection header received
0929 14:50:12.809771 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:12.809794 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:12.810214 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:12.810582 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:12.811377 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:12.812869 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:12.813785 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:12.814490 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:12.814656 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:12.817448 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:12.817454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:12.817470 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:12.817505 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:12.819763 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:12.820238 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:12.820244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:12.820250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:12.820398 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:12.820684 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:12.820688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:12.820689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:12.821224 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:12.821408 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:12.821793 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 14:50:12.822105 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":149,"threads_started":1}
I20260929 14:50:12.825065 14162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:12.825331 14162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:12.825392 14162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Initialized, Role: FOLLOWER
I20260929 14:50:12.825852 14162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:12.826021 14162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:12.826138 14162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:12.826261 14162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:12.827179 14162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:12.827527 14162 leader_election.cc:302] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fc90467b2f4f08b6a9bfe6efa3fd1c; no voters:
I20260929 14:50:12.827837 14162 leader_election.cc:288] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:50:12.828054 14167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:12.828471 14167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 LEADER]: Becoming Leader. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Running, Role: LEADER
I20260929 14:50:12.828943 14167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:12.829191 14162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:12.831024 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fc90467b2f4f08b6a9bfe6efa3fd1c. Latest consensus state: current_term: 1 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:12.831198 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:12.831159 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:12.831272 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
W20260929 14:50:12.834939 14182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:50:12.835031 14182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:50:12.835137 14181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:12.836071 14181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:12.842728 14181 catalog_manager.cc:1347] Generated new cluster ID: d1ca3b89298e4f1d83a81f5963ad209b
I20260929 14:50:12.842849 14181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:12.855863 14181 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:50:12.856959 14181 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:12.863418 14181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Generated new TSK 0
I20260929 14:50:12.864229 14181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:50:12.906998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:13.070340 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:13.070652 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:13.070700 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:13.074870 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:13.074942 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:13.075007 14188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:13.080524 14188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:13.081823 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:13.083058 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:13.094998 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.095088 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.095088 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:13.095511 14188 server_base.cc:1034] running on GCE node
I20260929 14:50:13.096138 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:13.096976 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:13.098178 14188 hybrid_clock.cc:648] HybridClock initialized: now 1790693413098148 us; error 49 us; skew 500 ppm
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:13.101280 14188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:13.102259 14188 webserver.cc:460] Webserver started at http://127.0.98.1:36357/ using document root <none> and password file <none>
I20260929 14:50:13.102887 14188 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:13.102973 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:13.103201 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:13.105417 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "6dff54dc0873484fa326b8b7d553eb5c"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "c33404560b591d045a852a7af06115ab"
server_key_iv: "07671922cef3f5041244a11638b5e4e4"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.106002 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6dff54dc0873484fa326b8b7d553eb5c"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "c33404560b591d045a852a7af06115ab"
server_key_iv: "07671922cef3f5041244a11638b5e4e4"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.110074 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:50:13.112807 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.114326 14188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:50:13.114473 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "6dff54dc0873484fa326b8b7d553eb5c"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "c33404560b591d045a852a7af06115ab"
server_key_iv: "07671922cef3f5041244a11638b5e4e4"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.114596 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:13.147354 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:13.150460 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:13.150668 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:13.151288 14188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:13.152163 14188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:13.152287 14188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.152359 14188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:13.152398 14188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.162864 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41319
I20260929 14:50:13.162951 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41319 every 8 connection(s)
I20260929 14:50:13.164003 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:13.165634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14188
I20260929 14:50:13.165798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:50:13.166136 392 external_mini_cluster.cc:1428] Setting key e91e2e7c2173372e70af0050da4b3f81
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:50:13.180593 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.166670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36341 (local address 127.0.98.62:37801)
0929 14:50:13.166856 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.166864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.167612 (+ 748us) server_negotiation.cc:407] Connection header received
0929 14:50:13.168741 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.168747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.168820 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.168926 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:13.170306 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.171193 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.172605 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.173078 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.173224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.176143 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:13.176149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:13.176155 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:13.176217 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:13.177895 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.178593 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.178596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.178598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.178691 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.179089 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.179093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.179095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.179398 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:13.179515 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.180295 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.180436 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:50:13.181617 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.166428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:36341)
0929 14:50:13.167396 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.167462 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.168498 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.169117 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.169125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.169561 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:13.170138 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.170153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.171368 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.171372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.172471 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.172483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.173257 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.173260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.173444 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.174207 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:13.174226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:13.175938 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:13.178056 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.178063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.178079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.178463 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.178786 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.178788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.178790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.178964 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.179561 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:13.179571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:13.179953 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.180233 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.180664 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":191,"threads_started":1}
I20260929 14:50:13.183296 14317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:13.183595 14317 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:13.184165 14317 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:13.185835 14126 ts_manager.cc:194] Registered new tserver with Master: 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319)
I20260929 14:50:13.186985 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:50:13.197438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:13.324672 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:13.324998 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:13.325078 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:13.329314 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:13.329396 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:13.329494 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:13.335549 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:13.337001 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:13.338282 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:13.350375 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.350355 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.350327 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:13.350862 14322 server_base.cc:1034] running on GCE node
I20260929 14:50:13.351397 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:13.352249 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:13.353473 14322 hybrid_clock.cc:648] HybridClock initialized: now 1790693413353448 us; error 38 us; skew 500 ppm
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:13.356802 14322 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:13.358110 14322 webserver.cc:460] Webserver started at http://127.0.98.2:45583/ using document root <none> and password file <none>
I20260929 14:50:13.358830 14322 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:13.358915 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:13.359135 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:13.361742 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd67a48869314277bf81865d86b6bc5d"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "e97310aff1292a7b4e95618c1e2bbf3b"
server_key_iv: "69601fadaa5d6e558c7101d8d62657a8"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.362682 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd67a48869314277bf81865d86b6bc5d"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "e97310aff1292a7b4e95618c1e2bbf3b"
server_key_iv: "69601fadaa5d6e558c7101d8d62657a8"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.368058 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 14:50:13.371299 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.373086 14322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:13.373245 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "bd67a48869314277bf81865d86b6bc5d"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "e97310aff1292a7b4e95618c1e2bbf3b"
server_key_iv: "69601fadaa5d6e558c7101d8d62657a8"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.373454 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:13.393546 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:13.397759 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:13.398173 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:13.398965 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:13.400028 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:13.400104 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.400184 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:13.400300 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 14:50:13.412535 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41119
I20260929 14:50:13.412559 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41119 every 8 connection(s)
I20260929 14:50:13.413707 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:13.415118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14322
I20260929 14:50:13.415254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:50:13.415580 392 external_mini_cluster.cc:1428] Setting key c3593a85db03005164bf4ba634019511
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:50:13.429031 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.415989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60061 (local address 127.0.98.62:37801)
0929 14:50:13.416232 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.416240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.417435 (+ 1195us) server_negotiation.cc:407] Connection header received
0929 14:50:13.418458 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.418461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.418524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.418639 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:13.419870 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.420480 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.421614 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.421836 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.421947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.424526 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:13.424535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:13.424537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:13.424559 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:13.426059 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.426907 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.426910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.426912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.426993 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.427565 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.427568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.427570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.427793 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:13.427891 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.428728 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.428847 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:50:13.430429 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.416437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:60061)
0929 14:50:13.417219 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.417301 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.418256 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.418812 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.418818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.419243 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:13.419722 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.419735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.420608 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.420612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.421491 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.421499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.421987 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.421991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.422171 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.422792 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:13.422808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:13.424361 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:13.426294 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.426306 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.426338 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.426764 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.427227 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.427230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.427232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.427422 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.427936 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:13.427942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:13.428437 (+ 495us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.428677 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.429124 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":158,"threads_started":1}
I20260929 14:50:13.432142 14451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:13.432508 14451 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:13.433187 14451 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:13.434595 14126 ts_manager.cc:194] Registered new tserver with Master: bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:13.435637 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:50:13.448354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:13.566090 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:13.566362 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:13.566406 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:13.570401 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:13.570459 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:13.570515 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:13.575626 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:13.576615 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:13.577649 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:13.587815 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.587814 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:13.587821 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:13.588162 14456 server_base.cc:1034] running on GCE node
I20260929 14:50:13.588619 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:13.589282 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:13.590478 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790693413590442 us; error 58 us; skew 500 ppm
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:13.593531 14456 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:13.594440 14456 webserver.cc:460] Webserver started at http://127.0.98.3:41357/ using document root <none> and password file <none>
I20260929 14:50:13.595041 14456 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:13.595116 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:13.595351 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:13.597610 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "40bd93703c39466890e845494fa4bd4a"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "ed8037a3cad3e6e2dbce8baf65fa925f"
server_key_iv: "777b37754f85015c31886b75a4d70a9c"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.598206 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40bd93703c39466890e845494fa4bd4a"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "ed8037a3cad3e6e2dbce8baf65fa925f"
server_key_iv: "777b37754f85015c31886b75a4d70a9c"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.602232 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:50:13.604868 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.606195 14456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:13.606299 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "40bd93703c39466890e845494fa4bd4a"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "ed8037a3cad3e6e2dbce8baf65fa925f"
server_key_iv: "777b37754f85015c31886b75a4d70a9c"
server_key_version: "encryptionkey@0"
I20260929 14:50:13.606384 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:13.628360 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:13.631639 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:13.631793 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:13.632385 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:13.633241 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:13.633293 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.633359 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:13.633395 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:13.644147 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46193
I20260929 14:50:13.644272 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46193 every 8 connection(s)
I20260929 14:50:13.645187 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:13.645740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14456
I20260929 14:50:13.645891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:50:13.646265 392 external_mini_cluster.cc:1428] Setting key c7aa1d89e0f9ccc8f1e4a1854fd0b875
Sep 29 14:50:13 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:13.661423 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.647318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58985 (local address 127.0.98.62:37801)
0929 14:50:13.647517 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.647523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.649058 (+ 1535us) server_negotiation.cc:407] Connection header received
0929 14:50:13.650637 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.650640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.650688 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.650780 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:13.652334 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.652870 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.654193 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.654421 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.654510 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.657407 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:13.657410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:13.657413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:13.657437 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:13.658915 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.659541 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.659544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.659548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.659661 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.660158 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.660161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.660163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.660401 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:13.660484 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.661207 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.661305 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 14:50:13.662582 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.647998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:58985)
0929 14:50:13.648877 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.648938 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.650352 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.651051 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.651059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.651501 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:13.652127 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.652144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.653011 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.653014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.654062 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.654074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.654600 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.654603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.654808 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.655479 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:13.655498 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:13.657214 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:13.659067 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.659075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.659108 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.659423 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.659795 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:13.659802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:13.659804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:13.660037 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:13.660500 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:13.660510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:13.660937 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.661133 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.661502 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":235,"threads_started":1}
I20260929 14:50:13.663959 14585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:13.664352 14585 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:13.664929 14585 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:13.666137 14126 ts_manager.cc:194] Registered new tserver with Master: 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193)
I20260929 14:50:13.666697 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58985
I20260929 14:50:13.673099 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:13.683674 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.675178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.0.98.62:37801)
0929 14:50:13.675390 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.675395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.675561 (+ 166us) server_negotiation.cc:407] Connection header received
0929 14:50:13.675769 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.675772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.675828 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.675919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:13.676815 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.677437 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.678573 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.678812 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.678926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.680520 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:13.680525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:13.680530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:13.680560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:13.681879 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.682329 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.682333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.682335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.682420 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:13.682700 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:13.682704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:13.682706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:13.682963 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:13.683116 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.683353 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.683482 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 14:50:13.687186 14126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:50:13.691185 14126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:50:13.708472 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.702191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.0.1:36788)
0929 14:50:13.703015 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.703026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.703194 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.704004 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.704006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.704020 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:13.704097 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.704101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705471 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.705474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.706660 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.706670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.707727 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.707730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.707878 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.707930 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.708069 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.708258 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":123,"threads_started":1}
I20260929 14:50:13.708724 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.701368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46193 (local address 127.0.0.1:41240)
0929 14:50:13.702384 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.702400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.702592 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.703224 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.703230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.703255 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:13.703434 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.703441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705191 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.705198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.707089 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.707102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.708120 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.708124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.708296 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.708324 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.708412 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.708524 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":161,"thread_start_us":116,"threads_started":1}
I20260929 14:50:13.708750 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.701478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.0.98.3:46193)
0929 14:50:13.701978 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.701993 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.702505 (+ 512us) server_negotiation.cc:407] Connection header received
0929 14:50:13.702805 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.702812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.703023 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.703606 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:13.703775 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705031 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.707256 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.707984 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.708090 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.708282 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.708469 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.708582 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":118,"threads_started":1}
I20260929 14:50:13.709481 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.701353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41319 (local address 127.0.0.1:37120)
0929 14:50:13.701763 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.701810 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.701948 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.703670 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.703673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.703696 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:13.703856 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.703869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705263 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.705266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.707089 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.707102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.708472 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.708475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.709167 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.709195 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.709268 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.709350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":136,"thread_start_us":180,"threads_started":1}
I20260929 14:50:13.709497 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.702503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.0.98.2:41119)
0929 14:50:13.702887 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.702899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.703303 (+ 404us) server_negotiation.cc:407] Connection header received
0929 14:50:13.703402 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.703408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.703633 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.703818 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:13.704259 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705043 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.706790 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.707564 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.707680 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.707860 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.709210 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.709346 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":180,"threads_started":1}
I20260929 14:50:13.710160 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.702638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.0.98.1:41319)
0929 14:50:13.703250 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.703257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.703279 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:50:13.703348 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.703352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.703523 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.703658 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:13.704007 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.705121 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.707267 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.708112 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.708255 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.708408 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.709876 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.710004 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":78,"threads_started":1}
I20260929 14:50:13.712244 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 (DEFAULT_TABLE table=test-table [id=61f2d60d53924919b8651f6d521c3114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:13.712244 14385 tablet_service.cc:1467] Processing CreateTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 (DEFAULT_TABLE table=test-table [id=61f2d60d53924919b8651f6d521c3114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:13.713018 14251 tablet_service.cc:1467] Processing CreateTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 (DEFAULT_TABLE table=test-table [id=61f2d60d53924919b8651f6d521c3114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:13.713397 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c7414f4daf4244a40dfe4a02a6bf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:13.713404 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c7414f4daf4244a40dfe4a02a6bf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:13.714035 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c7414f4daf4244a40dfe4a02a6bf44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:13.720961 14603 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Bootstrap starting.
I20260929 14:50:13.722131 14604 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap starting.
I20260929 14:50:13.722203 14602 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Bootstrap starting.
I20260929 14:50:13.724962 14604 tablet_bootstrap.cc:654] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:13.725445 14602 tablet_bootstrap.cc:654] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:13.726044 14604 log.cc:824] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:13.726217 14602 log.cc:824] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:13.726267 14603 tablet_bootstrap.cc:654] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:13.727572 14603 log.cc:824] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:13.728528 14604 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: No bootstrap required, opened a new log
I20260929 14:50:13.728528 14602 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: No bootstrap required, opened a new log
I20260929 14:50:13.728754 14604 ts_tablet_manager.cc:1397] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:13.728760 14602 ts_tablet_manager.cc:1397] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:13.729871 14603 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: No bootstrap required, opened a new log
I20260929 14:50:13.730089 14603 ts_tablet_manager.cc:1397] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260929 14:50:13.733234 14603 raft_consensus.cc:348] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.733479 14603 raft_consensus.cc:374] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:13.733542 14603 raft_consensus.cc:729] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bd93703c39466890e845494fa4bd4a, State: Initialized, Role: FOLLOWER
I20260929 14:50:13.733369 14602 raft_consensus.cc:348] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.733707 14602 raft_consensus.cc:374] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:13.733775 14602 raft_consensus.cc:729] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd67a48869314277bf81865d86b6bc5d, State: Initialized, Role: FOLLOWER
I20260929 14:50:13.733695 14604 raft_consensus.cc:348] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.733938 14604 raft_consensus.cc:374] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:13.733992 14604 raft_consensus.cc:729] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Initialized, Role: FOLLOWER
I20260929 14:50:13.734040 14603 consensus_queue.cc:260] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.734270 14602 consensus_queue.cc:260] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.734334 14604 consensus_queue.cc:260] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.735152 14585 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
I20260929 14:50:13.735368 14604 ts_tablet_manager.cc:1428] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:13.735368 14602 ts_tablet_manager.cc:1428] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:50:13.735764 14451 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
I20260929 14:50:13.736521 14317 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
I20260929 14:50:13.736979 14603 ts_tablet_manager.cc:1428] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260929 14:50:13.767942 14609 raft_consensus.cc:482] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:13.768311 14609 raft_consensus.cc:504] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.769748 14609 leader_election.cc:288] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:13.774492 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.770232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52047 (local address 127.0.98.2:41119)
0929 14:50:13.770433 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.770438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.770836 (+ 398us) server_negotiation.cc:407] Connection header received
0929 14:50:13.770984 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.770988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.771059 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.771149 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:13.771472 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.771972 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.773337 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.773859 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.773981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.774057 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.774265 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.774357 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 14:50:13.774521 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.770089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.98.1:52047)
0929 14:50:13.770642 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.770687 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.770840 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.771175 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.771178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.771192 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:13.771353 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.771360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.772128 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.772130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.773217 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.773226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.773986 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.773989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.774127 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.774159 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.774249 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.774346 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":326,"thread_start_us":89,"threads_started":1}
I20260929 14:50:13.774648 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.770258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37153 (local address 127.0.98.3:46193)
0929 14:50:13.770432 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.770438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.770709 (+ 271us) server_negotiation.cc:407] Connection header received
0929 14:50:13.770807 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.770811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.770870 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.770959 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:13.771325 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.771932 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.773397 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.773928 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.774063 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.774155 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.774393 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.774476 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 14:50:13.775211 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.770089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46193 (local address 127.0.98.1:37153)
0929 14:50:13.770586 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.770600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:13.770712 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:13.770993 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:13.770997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:13.771015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:13.771187 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.771195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.772061 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.772067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.773225 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:13.773238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.774072 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.774075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:13.774252 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.774272 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:50:13.774375 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:50:13.774476 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":40,"thread_start_us":153,"threads_started":1}
I20260929 14:50:13.775250 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c7414f4daf4244a40dfe4a02a6bf44" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d" is_pre_election: true
I20260929 14:50:13.775611 14405 raft_consensus.cc:2454] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 0.
I20260929 14:50:13.775951 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c7414f4daf4244a40dfe4a02a6bf44" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd93703c39466890e845494fa4bd4a" is_pre_election: true
I20260929 14:50:13.776249 14539 raft_consensus.cc:2454] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 0.
I20260929 14:50:13.776261 14207 leader_election.cc:302] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dff54dc0873484fa326b8b7d553eb5c, bd67a48869314277bf81865d86b6bc5d; no voters:
I20260929 14:50:13.776638 14609 raft_consensus.cc:2781] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:13.776734 14609 raft_consensus.cc:482] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:13.776768 14609 raft_consensus.cc:3037] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:13.778059 14609 raft_consensus.cc:504] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.778520 14609 leader_election.cc:288] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 election: Requested vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:13.778872 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c7414f4daf4244a40dfe4a02a6bf44" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd93703c39466890e845494fa4bd4a"
I20260929 14:50:13.779000 14539 raft_consensus.cc:3037] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:13.779037 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c7414f4daf4244a40dfe4a02a6bf44" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d"
I20260929 14:50:13.779218 14405 raft_consensus.cc:3037] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:13.780263 14539 raft_consensus.cc:2454] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 1.
I20260929 14:50:13.780588 14206 leader_election.cc:302] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bd93703c39466890e845494fa4bd4a, 6dff54dc0873484fa326b8b7d553eb5c; no voters:
I20260929 14:50:13.780617 14405 raft_consensus.cc:2454] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 1.
I20260929 14:50:13.780824 14609 raft_consensus.cc:2781] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:13.781112 14609 raft_consensus.cc:686] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 LEADER]: Becoming Leader. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Running, Role: LEADER
I20260929 14:50:13.781603 14609 consensus_queue.cc:237] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:13.785802 14124 catalog_manager.cc:5495] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dff54dc0873484fa326b8b7d553eb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } health_report { overall_health: UNKNOWN } } }
I20260929 14:50:13.813238 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.809197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.98.1:41319)
0929 14:50:13.809364 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:13.809369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:13.809384 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:50:13.809478 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:13.809480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:13.809545 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:13.809686 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:13.809993 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.810602 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.811560 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:13.811783 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:13.811896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:13.812252 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:13.812769 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:13.812913 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:13.812976 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 14:50:13.813048 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:50:13.820940 14539 raft_consensus.cc:1261] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Refusing update from remote peer 6dff54dc0873484fa326b8b7d553eb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:13.821002 14405 raft_consensus.cc:1261] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Refusing update from remote peer 6dff54dc0873484fa326b8b7d553eb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:13.821834 14613 consensus_queue.cc:1035] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:13.822057 14609 consensus_queue.cc:1035] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:13.840492 14126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:13.840682 14126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:13.843304 14126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Sending DeleteTablet for 3 replicas of tablet 21c7414f4daf4244a40dfe4a02a6bf44
I20260929 14:50:13.844089 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41240
I20260929 14:50:13.844384 14251 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37120
I20260929 14:50:13.844622 14626 tablet_replica.cc:330] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: stopping tablet replica
I20260929 14:50:13.844617 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36788
I20260929 14:50:13.844940 14626 raft_consensus.cc:2229] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:13.845158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14188
I20260929 14:50:13.845270 14626 raft_consensus.cc:2258] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:13.845486 14628 tablet_replica.cc:330] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: stopping tablet replica
I20260929 14:50:13.845930 14628 raft_consensus.cc:2229] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:13.846362 14628 raft_consensus.cc:2258] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:13.848001 14626 ts_tablet_manager.cc:1905] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:13.848278 14628 ts_tablet_manager.cc:1905] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:13.850584 14626 ts_tablet_manager.cc:1918] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:13.850703 14626 log.cc:1196] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/21c7414f4daf4244a40dfe4a02a6bf44
I20260929 14:50:13.850998 14626 ts_tablet_manager.cc:1939] T 21c7414f4daf4244a40dfe4a02a6bf44 P 40bd93703c39466890e845494fa4bd4a: Deleting consensus metadata
I20260929 14:50:13.851575 14628 ts_tablet_manager.cc:1918] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:13.851697 14628 log.cc:1196] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/21c7414f4daf4244a40dfe4a02a6bf44
I20260929 14:50:13.852042 14628 ts_tablet_manager.cc:1939] T 21c7414f4daf4244a40dfe4a02a6bf44 P bd67a48869314277bf81865d86b6bc5d: Deleting consensus metadata
I20260929 14:50:13.852298 14112 catalog_manager.cc:4841] TS 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193): tablet 21c7414f4daf4244a40dfe4a02a6bf44 (table test-table [id=61f2d60d53924919b8651f6d521c3114]) successfully deleted
I20260929 14:50:13.853560 14113 catalog_manager.cc:4841] TS bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119): tablet 21c7414f4daf4244a40dfe4a02a6bf44 (table test-table [id=61f2d60d53924919b8651f6d521c3114]) successfully deleted
W20260929 14:50:13.854671 14113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41319 (error 108)
I20260929 14:50:13.855073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14322
I20260929 14:50:13.856463 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:13.856122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41319 (local address 127.0.0.1:37146)
0929 14:50:13.856296 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:13.856374 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 14:50:13.856622 14113 catalog_manager.cc:4584] TS 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21c7414f4daf4244a40dfe4a02a6bf44: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41319: connect: Connection refused (error 111)
I20260929 14:50:13.861997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14456
I20260929 14:50:13.869153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14094
I20260929 14:50:13.877125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--webserver_interface=127.0.98.62
--webserver_port=34501
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:13.990258 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:13.990545 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:13.990586 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:13.994519 14630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:13.994590 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:13.994611 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:13.994626 14630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:13.994642 14630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:13.999915 14630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:14.001010 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:14.001966 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:14.016479 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:14.016482 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.016577 14630 server_base.cc:1034] running on GCE node
W20260929 14:50:14.016482 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.017663 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:14.019239 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:14.020483 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790693414020428 us; error 71 us; skew 500 ppm
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:14.024063 14630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:14.025434 14630 webserver.cc:460] Webserver started at http://127.0.98.62:34501/ using document root <none> and password file <none>
I20260929 14:50:14.026156 14630 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:14.026249 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:14.031603 14630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:14.034575 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.036410 14630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:14.036623 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c"
format_stamp: "Formatted at 2026-09-29 14:50:12 on dist-test-slave-5hd1"
server_key: "fc59d17d34b27ebd14614fd59a6ada55"
server_key_iv: "45d01d8b15c85a8edd282e49b9715f6a"
server_key_version: "encryptionkey@0"
I20260929 14:50:14.037601 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:14.069558 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:14.073854 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:14.074105 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:14.083755 14630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37801
I20260929 14:50:14.084139 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37801 every 8 connection(s)
I20260929 14:50:14.085335 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:14.085783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14630
I20260929 14:50:14.086300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41319
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=36357
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:14.091189 14699 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:50:14.093366 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap starting.
I20260929 14:50:14.115087 14699 log.cc:824] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:14.125149 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:14.125685 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap complete.
I20260929 14:50:14.130559 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:14.131052 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Initialized, Role: FOLLOWER
I20260929 14:50:14.131691 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:14.131824 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:14.131937 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:14.132066 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:50:14.134727 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:14.135191 14699 leader_election.cc:302] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fc90467b2f4f08b6a9bfe6efa3fd1c; no voters:
I20260929 14:50:14.135668 14699 leader_election.cc:288] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:50:14.135778 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:50:14.136027 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 LEADER]: Becoming Leader. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Running, Role: LEADER
I20260929 14:50:14.136425 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:14.137247 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:14.138108 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:14.138329 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:14.138706 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fc90467b2f4f08b6a9bfe6efa3fd1c. Latest consensus state: current_term: 2 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:14.138850 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:14.139183 14712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:14.144837 14712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:14.145531 14712 catalog_manager.cc:1259] Loaded cluster ID: d1ca3b89298e4f1d83a81f5963ad209b
I20260929 14:50:14.145632 14712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:14.147017 14712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 14:50:14.151569 14712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 14:50:14.157752 14712 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:14.158526 14712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Loaded TSK: 0
I20260929 14:50:14.159749 14712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:50:14.207286 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:14.207544 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:14.207587 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:14.211720 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:14.211796 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:14.211877 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:14.217111 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36357
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:14.218101 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:14.219070 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:14.229048 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:14.229169 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.229266 14701 server_base.cc:1034] running on GCE node
W20260929 14:50:14.229050 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.230110 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:14.231106 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:14.232302 14701 hybrid_clock.cc:648] HybridClock initialized: now 1790693414232256 us; error 57 us; skew 500 ppm
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:14.236950 14701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:14.238336 14701 webserver.cc:460] Webserver started at http://127.0.98.1:36357/ using document root <none> and password file <none>
I20260929 14:50:14.239339 14701 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:14.239464 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:14.245316 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:50:14.248474 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.250291 14701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:50:14.250455 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "6dff54dc0873484fa326b8b7d553eb5c"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "c33404560b591d045a852a7af06115ab"
server_key_iv: "07671922cef3f5041244a11638b5e4e4"
server_key_version: "encryptionkey@0"
I20260929 14:50:14.251204 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:14.276855 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:14.281169 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:14.281405 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:14.282167 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:14.283417 14743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:50:14.288369 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:50:14.288446 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 14:50:14.288523 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:50:14.290189 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:50:14.290252 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:50:14.290488 14743 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap starting.
I20260929 14:50:14.301723 14701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41319
I20260929 14:50:14.302205 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41319 every 8 connection(s)
I20260929 14:50:14.303074 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:14.303999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14701
I20260929 14:50:14.304555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41119
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=45583
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:14.311883 14743 log.cc:824] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:14.321617 14743 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:14.322113 14743 tablet_bootstrap.cc:492] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap complete.
I20260929 14:50:14.323570 14743 ts_tablet_manager.cc:1397] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260929 14:50:14.324347 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.306418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:36297)
0929 14:50:14.308001 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.308074 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.309438 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.311304 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.311313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.311836 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:14.312505 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.312527 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.314362 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.314366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.315280 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.315290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.316073 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.316077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.316318 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.317030 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:14.317047 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:14.318632 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:14.320883 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.320889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.320903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.321173 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.321536 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.321540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.321542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.321679 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.322387 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:14.322394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:14.322654 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.322849 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.323233 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"thread_start_us":196,"threads_started":1}
I20260929 14:50:14.324393 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.306902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36297 (local address 127.0.98.62:37801)
0929 14:50:14.307515 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.307546 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.308286 (+ 740us) server_negotiation.cc:407] Connection header received
0929 14:50:14.310042 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.310054 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.310456 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.310815 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:14.312719 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.314174 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.315423 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.315910 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.316096 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.318845 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:14.318851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:14.318864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:14.318892 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:14.320755 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.321313 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.321318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.321321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.321437 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.321812 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.321815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.321817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.322260 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:14.322420 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.322969 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.323246 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":127,"threads_started":1}
I20260929 14:50:14.327716 14851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:14.328008 14851 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:14.327543 14743 raft_consensus.cc:348] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:14.328205 14743 raft_consensus.cc:729] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Initialized, Role: FOLLOWER
I20260929 14:50:14.328743 14851 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:14.328713 14743 consensus_queue.cc:260] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } }
I20260929 14:50:14.329835 14743 ts_tablet_manager.cc:1428] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260929 14:50:14.331771 14663 ts_manager.cc:194] Registered new tserver with Master: 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319)
I20260929 14:50:14.335986 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36297
I20260929 14:50:14.339159 14851 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:50:14.348898 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.335404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41319 (local address 127.0.0.1:37152)
0929 14:50:14.336024 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.336076 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.336587 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.337244 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.337250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.337714 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:14.337863 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.337876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.339365 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.339369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.340742 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.340753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.341621 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.341637 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.341756 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.342440 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:14.342452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:14.344387 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:14.346658 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.346667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.346672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.347023 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.347385 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.347388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.347390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.347494 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.348110 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:14.348116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:14.348323 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.348472 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.348697 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":150,"threads_started":1}
I20260929 14:50:14.348927 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.335453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.0.98.1:41319)
0929 14:50:14.336362 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.336372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.336486 (+ 114us) server_negotiation.cc:407] Connection header received
0929 14:50:14.336759 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.336787 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.337037 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.337194 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:14.338088 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.339167 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.340924 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.341492 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.341621 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.344551 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:14.344560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:14.344612 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:14.344647 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:14.346521 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.347187 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.347194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.347195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.347284 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.347639 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.347642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.347643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.347974 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:14.348101 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.348588 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.348772 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":119,"threads_started":1}
I20260929 14:50:14.350283 14766 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c7414f4daf4244a40dfe4a02a6bf44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37152
I20260929 14:50:14.351397 14860 tablet_replica.cc:330] stopping tablet replica
I20260929 14:50:14.351781 14860 raft_consensus.cc:2229] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:14.352101 14860 raft_consensus.cc:2258] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:14.353330 14860 ts_tablet_manager.cc:1905] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:14.358038 14860 ts_tablet_manager.cc:1918] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:14.358139 14860 log.cc:1196] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/21c7414f4daf4244a40dfe4a02a6bf44
I20260929 14:50:14.358464 14860 ts_tablet_manager.cc:1939] T 21c7414f4daf4244a40dfe4a02a6bf44 P 6dff54dc0873484fa326b8b7d553eb5c: Deleting consensus metadata
I20260929 14:50:14.360581 14650 catalog_manager.cc:4841] TS 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319): tablet 21c7414f4daf4244a40dfe4a02a6bf44 (table test-table [id=61f2d60d53924919b8651f6d521c3114]) successfully deleted
W20260929 14:50:14.433734 14853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:14.434046 14853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:14.434127 14853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:14.438297 14853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:14.438375 14853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:14.438468 14853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:14.444046 14853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45583
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:14.445151 14853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:14.446148 14853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:14.456811 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:14.456861 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.457062 14853 server_base.cc:1034] running on GCE node
W20260929 14:50:14.457249 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.457645 14853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:14.458336 14853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:14.459525 14853 hybrid_clock.cc:648] HybridClock initialized: now 1790693414459507 us; error 35 us; skew 500 ppm
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:14.462397 14853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:14.463280 14853 webserver.cc:460] Webserver started at http://127.0.98.2:45583/ using document root <none> and password file <none>
I20260929 14:50:14.463824 14853 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:14.463928 14853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:14.467900 14853 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:14.470233 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.471676 14853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:14.471803 14853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "bd67a48869314277bf81865d86b6bc5d"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "e97310aff1292a7b4e95618c1e2bbf3b"
server_key_iv: "69601fadaa5d6e558c7101d8d62657a8"
server_key_version: "encryptionkey@0"
I20260929 14:50:14.472508 14853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:14.510650 14853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:14.513899 14853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:14.514101 14853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:14.514714 14853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:14.515540 14853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:14.515589 14853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.515630 14853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:14.515689 14853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.526305 14853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41119
I20260929 14:50:14.526414 14988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41119 every 8 connection(s)
I20260929 14:50:14.527420 14853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:14.528507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14853
I20260929 14:50:14.528993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46193
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=41357
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:14.546273 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.529909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55699 (local address 127.0.98.62:37801)
0929 14:50:14.530149 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.530157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.531340 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 14:50:14.532758 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.532762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.532819 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.532900 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:14.534820 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.535363 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.537438 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.537747 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.537848 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.541623 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:14.541630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:14.541636 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:14.541677 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:14.543744 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.544464 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.544468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.544470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.544567 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.544940 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.544952 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.544954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.545212 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:14.545321 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.545952 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.546099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 14:50:14.547370 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.530264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:55699)
0929 14:50:14.531152 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.531202 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.532472 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.533127 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.533139 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.533738 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:14.534602 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.534645 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.535512 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.535518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.537299 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.537314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.537953 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.537960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.538228 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.538974 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:14.539003 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:14.541412 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:14.543880 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.543886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.543904 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.544318 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.544664 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.544666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.544667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.544831 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.545331 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:14.545337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:14.545663 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.545864 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.546217 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":144,"threads_started":1}
I20260929 14:50:14.548933 14989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:14.549216 14989 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:14.550110 14989 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:14.552031 14663 ts_manager.cc:194] Registered new tserver with Master: bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:14.553359 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55699
W20260929 14:50:14.649819 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:14.650092 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:14.650136 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:14.654244 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:14.654321 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:14.654382 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:14.660064 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41357
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:14.661206 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:14.662264 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:14.672573 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:14.672580 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.672952 14992 server_base.cc:1034] running on GCE node
W20260929 14:50:14.672670 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:14.673435 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:14.674124 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:14.675295 14992 hybrid_clock.cc:648] HybridClock initialized: now 1790693414675273 us; error 43 us; skew 500 ppm
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:14.678259 14992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:14.679225 14992 webserver.cc:460] Webserver started at http://127.0.98.3:41357/ using document root <none> and password file <none>
I20260929 14:50:14.679801 14992 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:14.679908 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:14.684078 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:14.686484 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.687937 14992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:14.688094 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "40bd93703c39466890e845494fa4bd4a"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "ed8037a3cad3e6e2dbce8baf65fa925f"
server_key_iv: "777b37754f85015c31886b75a4d70a9c"
server_key_version: "encryptionkey@0"
I20260929 14:50:14.688722 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:14.723378 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:14.726861 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:14.727083 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:14.727720 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:14.728622 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:14.728695 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.728766 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:14.728801 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:14.739344 14992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46193
I20260929 14:50:14.739485 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46193 every 8 connection(s)
I20260929 14:50:14.740568 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:14.750947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 14992
Sep 29 14:50:14 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:14.756891 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.743110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49591 (local address 127.0.98.62:37801)
0929 14:50:14.743334 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.743339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.744586 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 14:50:14.745609 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.745612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.745665 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.745751 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:14.747165 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.747704 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.749101 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.749311 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.749401 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.752815 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:14.752819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:14.752822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:14.752850 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:14.754503 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.755022 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.755025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.755027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.755108 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.755577 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.755579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.755580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.755734 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:14.755852 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.756626 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.756771 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 14:50:14.758114 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.743536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:49591)
0929 14:50:14.744388 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.744441 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.745378 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.745960 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.745967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.746423 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:14.746995 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.747010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.747858 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.747865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.748970 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.748980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.749488 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.749492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.749656 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.750375 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:14.750393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:14.752647 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:14.754625 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.754631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.754650 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.754903 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.755220 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:14.755223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:14.755225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:14.755444 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:14.755872 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:14.755878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:14.756314 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.756568 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.757022 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":124,"threads_started":1}
I20260929 14:50:14.759563 15121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37801
I20260929 14:50:14.759855 15121 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:14.760593 15121 heartbeater.cc:511] Master 127.0.98.62:37801 requested a full tablet report, sending...
I20260929 14:50:14.760999 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.751808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.0.98.62:37801)
0929 14:50:14.752069 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.752073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.752206 (+ 133us) server_negotiation.cc:407] Connection header received
0929 14:50:14.752366 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.752369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.752415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.752526 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:14.753373 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.754185 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.755078 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.755357 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.755434 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.757200 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:14.757204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:14.757208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:14.757237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:14.759060 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.759484 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.759487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.759489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.759581 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:14.759962 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:14.759968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:14.759970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:14.760133 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:14.760404 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.760700 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.760821 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260929 14:50:14.761914 14663 ts_manager.cc:194] Registered new tserver with Master: 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193)
I20260929 14:50:14.762509 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49591
I20260929 14:50:14.763175 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:14.767987 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.763968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.0.98.62:37801)
0929 14:50:14.764104 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.764108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.764358 (+ 250us) server_negotiation.cc:407] Connection header received
0929 14:50:14.764547 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.764550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.764663 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.764780 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:14.765023 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.765557 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.766455 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.766701 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.766809 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.767138 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:14.767582 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:14.767726 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.767767 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.767835 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:50:14.768761 14662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:50:14.772619 14662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:50:14.782440 14766 tablet_service.cc:1467] Processing CreateTablet for tablet 36093c5c4b404c479fef50950ab879ac (DEFAULT_TABLE table=test-table [id=0009b7edff4444c99da2b7b33b0f9ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:14.783159 14766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36093c5c4b404c479fef50950ab879ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:14.786485 14743 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap starting.
I20260929 14:50:14.787358 14743 tablet_bootstrap.cc:654] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:14.787333 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.781067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46193 (local address 127.0.0.1:41244)
0929 14:50:14.781233 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.781252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.781343 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.782385 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.782388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.782405 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:14.782519 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.782524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.784244 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.784247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.785635 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.785645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.786843 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.786847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.786954 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.787009 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.787060 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.787157 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":26}
I20260929 14:50:14.788108 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.781309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.98.3:46193)
0929 14:50:14.781814 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.781830 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.781861 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:50:14.781950 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.781956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.782202 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.782403 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:14.782701 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.784039 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.785791 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.786707 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.787687 (+ 980us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.787811 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.787858 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.787946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":192,"threads_started":1}
I20260929 14:50:14.788439 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.781687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.0.98.2:41119)
0929 14:50:14.782049 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.782068 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.782971 (+ 903us) server_negotiation.cc:407] Connection header received
0929 14:50:14.783079 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.783084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.783260 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.783435 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:14.783652 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.784788 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.786524 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.787359 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.787470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.787658 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.788140 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.788286 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":149,"threads_started":1}
I20260929 14:50:14.788764 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.781357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.0.1:36790)
0929 14:50:14.782720 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.782736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.782848 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.783409 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.783412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.783430 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:14.783534 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.783540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.784964 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.784968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.786366 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.786375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.787747 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.787755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.787947 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.787981 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.788092 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.788402 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"mutex_wait_us":24,"thread_start_us":117,"threads_started":1}
I20260929 14:50:14.789306 14743 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: No bootstrap required, opened a new log
I20260929 14:50:14.789422 14743 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:14.789963 14743 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.790112 14743 raft_consensus.cc:374] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:14.790150 14743 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Initialized, Role: FOLLOWER
I20260929 14:50:14.790308 14743 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.790585 14743 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:50:14.791898 15055 tablet_service.cc:1467] Processing CreateTablet for tablet 36093c5c4b404c479fef50950ab879ac (DEFAULT_TABLE table=test-table [id=0009b7edff4444c99da2b7b33b0f9ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:14.792160 14923 tablet_service.cc:1467] Processing CreateTablet for tablet 36093c5c4b404c479fef50950ab879ac (DEFAULT_TABLE table=test-table [id=0009b7edff4444c99da2b7b33b0f9ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:14.793402 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36093c5c4b404c479fef50950ab879ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:14.793465 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36093c5c4b404c479fef50950ab879ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:14.800861 15134 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Bootstrap starting.
I20260929 14:50:14.800945 15135 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Bootstrap starting.
I20260929 14:50:14.803563 15134 tablet_bootstrap.cc:654] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:14.803635 15135 tablet_bootstrap.cc:654] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:14.804585 15135 log.cc:824] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:14.804782 15134 log.cc:824] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Log is configured to *not* fsync() on all Append() calls
W20260929 14:50:14.804828 14852 tablet.cc:2369] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:50:14.806938 15135 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: No bootstrap required, opened a new log
I20260929 14:50:14.807065 15134 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: No bootstrap required, opened a new log
I20260929 14:50:14.807166 15135 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:50:14.807303 15134 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:50:14.810453 15134 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.810451 15135 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.810750 15134 raft_consensus.cc:374] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:14.810740 15135 raft_consensus.cc:374] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:14.810815 15135 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bd93703c39466890e845494fa4bd4a, State: Initialized, Role: FOLLOWER
I20260929 14:50:14.810819 15134 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd67a48869314277bf81865d86b6bc5d, State: Initialized, Role: FOLLOWER
I20260929 14:50:14.811307 15134 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.811314 15135 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.812534 14989 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
I20260929 14:50:14.812531 15121 heartbeater.cc:503] Master 127.0.98.62:37801 was elected leader, sending a full tablet report...
I20260929 14:50:14.812570 15135 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 14:50:14.813410 15134 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:50:14.884070 14857 raft_consensus.cc:482] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:14.884712 14857 raft_consensus.cc:504] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.886128 14857 leader_election.cc:288] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:14.890724 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.886638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35517 (local address 127.0.98.3:46193)
0929 14:50:14.886781 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.886786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.887163 (+ 377us) server_negotiation.cc:407] Connection header received
0929 14:50:14.887331 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.887334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.887382 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.887473 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:14.887757 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.888345 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.889640 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.890058 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.890143 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.890197 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.890500 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.890572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:50:14.890769 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.886437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46193 (local address 127.0.98.1:35517)
0929 14:50:14.887018 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.887031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.887192 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.887500 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.887502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.887517 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:14.887619 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.887647 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.888488 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.888491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.889482 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.889491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.890217 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.890221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.890317 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.890381 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.890463 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.890552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":174,"thread_start_us":111,"threads_started":1}
I20260929 14:50:14.891026 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.886628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44787 (local address 127.0.98.2:41119)
0929 14:50:14.886841 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.886846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.886952 (+ 106us) server_negotiation.cc:407] Connection header received
0929 14:50:14.887144 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.887147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.887208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.887299 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:14.887688 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.888326 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.889603 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.890359 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.890484 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.890551 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.890779 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.890871 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:50:14.891044 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.886438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.98.1:44787)
0929 14:50:14.886854 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.886867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:14.886988 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:14.887329 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:14.887332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:14.887353 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:14.887560 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.887569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.888440 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.888446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.889475 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:14.889484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.890522 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.890525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:14.890628 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.890657 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:50:14.890742 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:50:14.890840 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":9,"thread_start_us":101,"threads_started":1}
I20260929 14:50:14.891438 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd93703c39466890e845494fa4bd4a" is_pre_election: true
I20260929 14:50:14.891794 15075 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 0.
I20260929 14:50:14.891793 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d" is_pre_election: true
I20260929 14:50:14.892127 14943 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 0.
I20260929 14:50:14.892382 14739 leader_election.cc:302] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bd93703c39466890e845494fa4bd4a, 6dff54dc0873484fa326b8b7d553eb5c; no voters:
I20260929 14:50:14.892753 14857 raft_consensus.cc:2781] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:14.892850 14857 raft_consensus.cc:482] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:14.892894 14857 raft_consensus.cc:3037] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:14.894084 14857 raft_consensus.cc:504] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.894481 14857 leader_election.cc:288] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 election: Requested vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:14.894841 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bd93703c39466890e845494fa4bd4a"
I20260929 14:50:14.894989 15075 raft_consensus.cc:3037] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:14.894999 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d"
I20260929 14:50:14.895226 14943 raft_consensus.cc:3037] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:14.896087 15075 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 1.
I20260929 14:50:14.896478 14739 leader_election.cc:302] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bd93703c39466890e845494fa4bd4a, 6dff54dc0873484fa326b8b7d553eb5c; no voters:
I20260929 14:50:14.896658 14857 raft_consensus.cc:2781] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:14.896654 14943 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 1.
I20260929 14:50:14.896885 14857 raft_consensus.cc:686] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 LEADER]: Becoming Leader. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Running, Role: LEADER
I20260929 14:50:14.897200 14857 consensus_queue.cc:237] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:14.900940 14662 catalog_manager.cc:5495] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6dff54dc0873484fa326b8b7d553eb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } health_report { overall_health: HEALTHY } } }
I20260929 14:50:14.915058 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.910869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.98.1:41319)
0929 14:50:14.911237 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:14.911245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:14.911259 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:50:14.911346 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:14.911348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:14.911407 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:14.911524 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:14.911846 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.912517 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.913377 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:14.913607 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:14.913691 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:14.913996 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:14.914588 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:14.914696 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:14.914769 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 14:50:14.914825 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":165,"threads_started":1}
I20260929 14:50:14.919602 14943 raft_consensus.cc:1261] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Refusing update from remote peer 6dff54dc0873484fa326b8b7d553eb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:14.919858 15075 raft_consensus.cc:1261] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Refusing update from remote peer 6dff54dc0873484fa326b8b7d553eb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:14.920346 15142 consensus_queue.cc:1035] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:14.920622 14857 consensus_queue.cc:1035] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:14.929656 15147 mvcc.cc:205] Tried to move back new op lower bound from 7334680227503239168 to 7334680227422851072. Current Snapshot: MvccSnapshot[applied={T|T < 7334680227503239168}]
I20260929 14:50:14.936976 14661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:14.937227 14661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:14.940335 14661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Sending DeleteTablet for 3 replicas of tablet 36093c5c4b404c479fef50950ab879ac
I20260929 14:50:14.941326 14766 tablet_service.cc:1514] Processing DeleteTablet for tablet 36093c5c4b404c479fef50950ab879ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37152
I20260929 14:50:14.941318 14923 tablet_service.cc:1514] Processing DeleteTablet for tablet 36093c5c4b404c479fef50950ab879ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36790
I20260929 14:50:14.941293 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet 36093c5c4b404c479fef50950ab879ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41244
I20260929 14:50:14.941895 15156 tablet_replica.cc:330] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: stopping tablet replica
I20260929 14:50:14.942023 15155 tablet_replica.cc:330] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: stopping tablet replica
I20260929 14:50:14.942230 15155 raft_consensus.cc:2229] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:50:14.942344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14701
I20260929 14:50:14.943243 15157 tablet_replica.cc:330] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: stopping tablet replica
I20260929 14:50:14.943656 15157 raft_consensus.cc:2229] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:14.943986 15157 raft_consensus.cc:2258] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:14.945988 15157 ts_tablet_manager.cc:1905] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:14.948701 15157 ts_tablet_manager.cc:1918] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:14.948807 15157 log.cc:1196] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/36093c5c4b404c479fef50950ab879ac
I20260929 14:50:14.949044 15156 raft_consensus.cc:2229] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:14.949199 15157 ts_tablet_manager.cc:1939] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Deleting consensus metadata
I20260929 14:50:14.949388 15156 raft_consensus.cc:2258] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:50:14.950364 14650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41319 (error 108)
I20260929 14:50:14.950455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14853
I20260929 14:50:14.951560 14649 catalog_manager.cc:4841] TS 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193): tablet 36093c5c4b404c479fef50950ab879ac (table test-table [id=0009b7edff4444c99da2b7b33b0f9ff2]) successfully deleted
I20260929 14:50:14.952030 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.951660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41319 (local address 127.0.0.1:37158)
0929 14:50:14.951855 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.951925 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 14:50:14.952325 14650 catalog_manager.cc:4584] TS 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36093c5c4b404c479fef50950ab879ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41319: connect: Connection refused (error 111)
I20260929 14:50:14.958374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14992
I20260929 14:50:14.959450 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:14.958539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.0.1:36800)
0929 14:50:14.959195 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:14.959261 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563}
I20260929 14:50:14.972635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 14630
I20260929 14:50:14.980053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--webserver_interface=127.0.98.62
--webserver_port=34501
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:15.113670 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:15.114001 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:15.114046 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:15.118261 15159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:15.118338 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:15.118358 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:15.118376 15159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:15.118392 15159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:15.124046 15159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:15.126186 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:15.127524 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:15.138150 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:15.138262 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:15.138268 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.138450 15159 server_base.cc:1034] running on GCE node
I20260929 14:50:15.139635 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:15.141079 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:15.142306 15159 hybrid_clock.cc:648] HybridClock initialized: now 1790693415142273 us; error 53 us; skew 500 ppm
Sep 29 14:50:15 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:15.152694 15159 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:15.154094 15159 webserver.cc:460] Webserver started at http://127.0.98.62:34501/ using document root <none> and password file <none>
I20260929 14:50:15.154829 15159 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:15.154901 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:15.159732 15159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:15.162576 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.164569 15159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:15.164750 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c"
format_stamp: "Formatted at 2026-09-29 14:50:12 on dist-test-slave-5hd1"
server_key: "fc59d17d34b27ebd14614fd59a6ada55"
server_key_iv: "45d01d8b15c85a8edd282e49b9715f6a"
server_key_version: "encryptionkey@0"
I20260929 14:50:15.165442 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:15.184639 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:15.189109 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:15.189354 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:15.197332 15159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37801
I20260929 14:50:15.197443 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37801 every 8 connection(s)
I20260929 14:50:15.198504 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:15.204424 15227 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:50:15.206585 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap starting.
I20260929 14:50:15.208389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15159
I20260929 14:50:15.208825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41319
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=36357
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:15.228559 15227 log.cc:824] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:15.241353 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:15.241871 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c: Bootstrap complete.
I20260929 14:50:15.247294 15227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:15.247812 15227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Initialized, Role: FOLLOWER
I20260929 14:50:15.248572 15227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:15.248711 15227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:15.248818 15227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:15.248956 15227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:50:15.251540 15227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:15.251977 15227 leader_election.cc:302] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fc90467b2f4f08b6a9bfe6efa3fd1c; no voters:
I20260929 14:50:15.252434 15227 leader_election.cc:288] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:50:15.252543 15231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:50:15.252763 15231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [term 3 LEADER]: Becoming Leader. State: Replica: 89fc90467b2f4f08b6a9bfe6efa3fd1c, State: Running, Role: LEADER
I20260929 14:50:15.253121 15231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } }
I20260929 14:50:15.253993 15227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:15.254235 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:15.254375 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:15.255081 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fc90467b2f4f08b6a9bfe6efa3fd1c. Latest consensus state: current_term: 3 leader_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc90467b2f4f08b6a9bfe6efa3fd1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37801 } } }
I20260929 14:50:15.255214 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc90467b2f4f08b6a9bfe6efa3fd1c [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:15.255932 15235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:15.262668 15235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:15.263968 15235 catalog_manager.cc:1259] Loaded cluster ID: d1ca3b89298e4f1d83a81f5963ad209b
I20260929 14:50:15.264053 15235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 14:50:15.265161 15235 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe5eddc59ad google::LogMessage::Fail() at ??:0
@ 0x7fe5eddc888a google::LogMessage::SendToLog() at ??:0
@ 0x7fe5eddc54de google::LogMessage::Flush() at ??:0
@ 0x7fe5eddc6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5f8704fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe5f8705d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5f8702579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe5f8746640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe5f00173d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe5f0017c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe5f00193f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe5f0008512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe5f164c6db start_thread at ??:0
@ 0x7fe5ec8c361f clone at ??:0
W20260929 14:50:15.381994 15229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:15.382562 15229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:15.382647 15229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:15.389261 15229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:15.389358 15229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:15.389432 15229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:15.395933 15229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36357
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:15.397411 15229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:15.398699 15229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:15.410936 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:15.411006 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.411340 15229 server_base.cc:1034] running on GCE node
W20260929 14:50:15.411434 15271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.412048 15229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:15.412911 15229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:15.414171 15229 hybrid_clock.cc:648] HybridClock initialized: now 1790693415414125 us; error 61 us; skew 500 ppm
Sep 29 14:50:15 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:15.417937 15229 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:15.419315 15229 webserver.cc:460] Webserver started at http://127.0.98.1:36357/ using document root <none> and password file <none>
I20260929 14:50:15.420229 15229 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:15.420337 15229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:15.426160 15229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:50:15.429909 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.431897 15229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:50:15.432094 15229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "6dff54dc0873484fa326b8b7d553eb5c"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "c33404560b591d045a852a7af06115ab"
server_key_iv: "07671922cef3f5041244a11638b5e4e4"
server_key_version: "encryptionkey@0"
I20260929 14:50:15.433101 15229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:15.455801 15229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:15.460357 15229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:15.460599 15229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:15.461463 15229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:15.463145 15285 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:50:15.467621 15229 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:50:15.467715 15229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 14:50:15.467803 15229 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:50:15.470762 15229 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:50:15.470844 15229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 14:50:15.471215 15285 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap starting.
I20260929 14:50:15.485705 15229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41319
I20260929 14:50:15.485822 15392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41319 every 8 connection(s)
I20260929 14:50:15.487308 15229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:15.491485 15285 log.cc:824] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:15.497172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15229
I20260929 14:50:15.497648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41119
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=45583
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:15.499607 15285 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:15.499985 15285 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Bootstrap complete.
I20260929 14:50:15.501302 15285 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.006s
I20260929 14:50:15.503865 15285 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:15.504365 15285 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Initialized, Role: FOLLOWER
I20260929 14:50:15.504940 15285 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:15.505789 15285 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260929 14:50:15.579456 15395 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:50:15.490362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:53011)
0929 14:50:15.491904 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.491972 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:15.493257 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:15.578539 (+ 85282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1261,"thread_start_us":178,"threads_started":1}
W20260929 14:50:15.580047 15393 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37801 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37801: Client connection negotiation failed: client connection to 127.0.98.62:37801: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:50:15.581033 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.580740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:38725)
0929 14:50:15.580910 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.580979 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 14:50:15.581743 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.581529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:55097)
0929 14:50:15.581640 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.581688 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 14:50:15.581903 15393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:50:15.655483 15397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:15.655889 15397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:15.655982 15397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:15.660488 15397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:15.660627 15397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:15.660746 15397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:15.666945 15397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45583
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:15.668435 15397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:15.669725 15397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:15.681644 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:15.681644 15403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.681789 15397 server_base.cc:1034] running on GCE node
W20260929 14:50:15.681667 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.682803 15397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:15.683637 15397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:15.684847 15397 hybrid_clock.cc:648] HybridClock initialized: now 1790693415684809 us; error 59 us; skew 500 ppm
Sep 29 14:50:15 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:15.688453 15397 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:15.689683 15397 webserver.cc:460] Webserver started at http://127.0.98.2:45583/ using document root <none> and password file <none>
I20260929 14:50:15.690415 15397 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:15.690515 15397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:15.695310 15397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:50:15.698192 15413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.699796 15397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:15.699954 15397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "bd67a48869314277bf81865d86b6bc5d"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "e97310aff1292a7b4e95618c1e2bbf3b"
server_key_iv: "69601fadaa5d6e558c7101d8d62657a8"
server_key_version: "encryptionkey@0"
I20260929 14:50:15.700726 15397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:15.717159 15397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:15.720985 15397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:15.721225 15397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:15.721942 15397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:15.723279 15420 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:50:15.727734 15397 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:50:15.727849 15397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 14:50:15.727950 15397 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:50:15.730144 15397 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:50:15.730223 15397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 14:50:15.730640 15420 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Bootstrap starting.
I20260929 14:50:15.736353 15420 log.cc:824] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:15.743574 15397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41119
I20260929 14:50:15.743644 15528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41119 every 8 connection(s)
I20260929 14:50:15.744777 15397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:15.745359 15420 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:15.746232 15420 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Bootstrap complete.
I20260929 14:50:15.746608 15420 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260929 14:50:15.748134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15397
I20260929 14:50:15.748690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46193
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37801
--webserver_port=41357
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:15.749084 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.747181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:41581)
0929 14:50:15.747761 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.747946 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
W20260929 14:50:15.749696 15529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37801 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37801: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
I20260929 14:50:15.749728 15420 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:15.750252 15420 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd67a48869314277bf81865d86b6bc5d, State: Initialized, Role: FOLLOWER
I20260929 14:50:15.750613 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.750214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:43361)
0929 14:50:15.750456 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.750528 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 14:50:15.750766 15420 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:15.751509 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.751274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:55063)
0929 14:50:15.751400 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.751454 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:50:15.751724 15420 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 14:50:15.751754 15529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:50:15.752307 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.752065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:33339)
0929 14:50:15.752210 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.752264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:50:15.753093 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.752782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:59677)
0929 14:50:15.752934 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.753002 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 14:50:15.878203 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:15.878497 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:15.878546 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:15.883373 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:15.883459 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:15.883527 15532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:15.889423 15532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41357
--tserver_master_addrs=127.0.98.62:37801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:15.890691 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:15.891886 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:15.903234 15538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.903499 15532 server_base.cc:1034] running on GCE node
W20260929 14:50:15.903201 15541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:15.903201 15539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:15.904072 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:15.904945 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:15.906136 15532 hybrid_clock.cc:648] HybridClock initialized: now 1790693415906113 us; error 42 us; skew 500 ppm
Sep 29 14:50:15 dist-test-slave-5hd1 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:15.909166 15532 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:15.910177 15532 webserver.cc:460] Webserver started at http://127.0.98.3:41357/ using document root <none> and password file <none>
I20260929 14:50:15.910879 15532 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:15.910995 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:15.915416 15532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:15.918081 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.919526 15532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:50:15.919678 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "40bd93703c39466890e845494fa4bd4a"
format_stamp: "Formatted at 2026-09-29 14:50:13 on dist-test-slave-5hd1"
server_key: "ed8037a3cad3e6e2dbce8baf65fa925f"
server_key_iv: "777b37754f85015c31886b75a4d70a9c"
server_key_version: "encryptionkey@0"
I20260929 14:50:15.920338 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:15.932461 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:15.935840 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:15.936057 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:15.936851 15532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:15.937842 15532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:15.937922 15532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.937987 15532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:15.938046 15532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:15.949328 15532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46193
I20260929 14:50:15.949388 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46193 every 8 connection(s)
I20260929 14:50:15.950405 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:15.954833 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.953012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:43955)
0929 14:50:15.953796 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.953941 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":124,"threads_started":1}
W20260929 14:50:15.955361 15662 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37801 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37801: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
I20260929 14:50:15.956044 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.955783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:54585)
0929 14:50:15.955929 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.955986 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:50:15.956898 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:15.956666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:36491)
0929 14:50:15.956798 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:15.956851 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 14:50:15.957168 15662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:50:15.958225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15532
I20260929 14:50:16.583370 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.582675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:35941)
0929 14:50:16.583081 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.583178 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":81,"threads_started":1}
I20260929 14:50:16.754659 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.754119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:45789)
0929 14:50:16.754460 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.754562 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260929 14:50:16.874198 15668 raft_consensus.cc:482] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:16.874467 15668 raft_consensus.cc:504] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:16.875658 15668 leader_election.cc:288] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
Sep 29 14:50:16 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 14:50:16 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 14:50:16.892565 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.875855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46193 (local address 127.0.98.1:39003)
0929 14:50:16.876106 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.876124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:16.876394 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:16.878443 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:16.878447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:16.878920 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:16.879429 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:16.879448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.880987 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.880991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:16.882153 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:16.882165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.882971 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.882977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:16.883164 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:16.884085 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:16.884112 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:16.886015 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:16.888998 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:16.889002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:16.889004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:16.889255 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:16.889821 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:16.889825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:16.889827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:16.890039 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:16.891193 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:16.891201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:16.891636 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 14:50:16.891876 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 14:50:16.892321 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":167,"spinlock_wait_cycles":6528}
I20260929 14:50:16.892596 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.876100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41119 (local address 127.0.98.1:56097)
0929 14:50:16.876631 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.876645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:16.876762 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:16.878282 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:16.878291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:16.878976 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:16.879426 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:16.879444 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.881009 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.881012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:16.882147 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:16.882158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.883061 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.883065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:16.883252 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:16.884840 (+ 1588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:16.884854 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:16.886390 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:16.888757 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:16.888764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:16.888778 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:16.889162 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:16.889821 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:16.889825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:16.889827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:16.890039 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:16.891193 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:16.891201 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:16.891636 (+ 435us) client_negotiation.cc:769] Sending connection context
0929 14:50:16.891876 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 14:50:16.892381 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":355,"spinlock_wait_cycles":63488,"thread_start_us":125,"threads_started":1}
I20260929 14:50:16.893086 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.876088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39003 (local address 127.0.98.3:46193)
0929 14:50:16.876702 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:16.876720 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:16.876772 (+ 52us) server_negotiation.cc:407] Connection header received
0929 14:50:16.877475 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:16.877481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:16.878088 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:16.878454 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:16.879655 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.880817 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.882331 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.882817 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.882947 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:16.886308 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:16.886314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:16.886336 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:16.886365 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:16.888824 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:16.889410 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:16.889418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:16.889426 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:16.889671 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:16.890186 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:16.890191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:16.890193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:16.891015 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:16.891193 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:16.892110 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 14:50:16.892565 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":126,"threads_started":1}
I20260929 14:50:16.893106 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.876237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56097 (local address 127.0.98.2:41119)
0929 14:50:16.876643 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:16.876659 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:16.876742 (+ 83us) server_negotiation.cc:407] Connection header received
0929 14:50:16.877108 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:16.877114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:16.877571 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:16.878127 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:16.879625 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.880817 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.882296 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:16.882896 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:16.883088 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:16.886591 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:16.886597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:16.886608 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:16.886641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:16.888603 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:16.889411 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:16.889418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:16.889426 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:16.889671 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:16.890186 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:16.890191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:16.890193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:16.891015 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:16.891193 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:16.892110 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 14:50:16.892565 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260929 14:50:16.894556 15483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d" is_pre_election: true
I20260929 14:50:16.894982 15483 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 1.
W20260929 14:50:16.895973 15281 leader_election.cc:341] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193): Not found: Tablet not found: 36093c5c4b404c479fef50950ab879ac
I20260929 14:50:16.896248 15282 leader_election.cc:302] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dff54dc0873484fa326b8b7d553eb5c, bd67a48869314277bf81865d86b6bc5d; no voters: 40bd93703c39466890e845494fa4bd4a
I20260929 14:50:16.896653 15668 raft_consensus.cc:2781] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:50:16.896762 15668 raft_consensus.cc:482] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:16.896816 15668 raft_consensus.cc:3037] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:50:16.898968 15668 raft_consensus.cc:504] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:16.899431 15668 leader_election.cc:288] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 election: Requested vote from peers 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193), bd67a48869314277bf81865d86b6bc5d (127.0.98.2:41119)
I20260929 14:50:16.900070 15483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36093c5c4b404c479fef50950ab879ac" candidate_uuid: "6dff54dc0873484fa326b8b7d553eb5c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bd67a48869314277bf81865d86b6bc5d"
W20260929 14:50:16.900153 15281 leader_election.cc:341] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193): Not found: Tablet not found: 36093c5c4b404c479fef50950ab879ac
I20260929 14:50:16.900337 15483 raft_consensus.cc:3037] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:50:16.903561 15483 raft_consensus.cc:2454] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dff54dc0873484fa326b8b7d553eb5c in term 2.
I20260929 14:50:16.904124 15282 leader_election.cc:302] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dff54dc0873484fa326b8b7d553eb5c, bd67a48869314277bf81865d86b6bc5d; no voters: 40bd93703c39466890e845494fa4bd4a
I20260929 14:50:16.904449 15668 raft_consensus.cc:2781] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:50:16.904717 15668 raft_consensus.cc:686] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [term 2 LEADER]: Becoming Leader. State: Replica: 6dff54dc0873484fa326b8b7d553eb5c, State: Running, Role: LEADER
I20260929 14:50:16.905124 15668 consensus_queue.cc:237] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:16.906045 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.905527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:56707)
0929 14:50:16.905856 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.905943 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260929 14:50:16.958551 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:16.957986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:53947)
0929 14:50:16.958359 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:16.958437 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
W20260929 14:50:17.411945 15281 consensus_peers.cc:487] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c -> Peer 40bd93703c39466890e845494fa4bd4a (127.0.98.3:46193): Couldn't send request to peer 40bd93703c39466890e845494fa4bd4a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36093c5c4b404c479fef50950ab879ac. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:50:17.449002 15483 raft_consensus.cc:1261] T 36093c5c4b404c479fef50950ab879ac P bd67a48869314277bf81865d86b6bc5d [term 2 FOLLOWER]: Refusing update from remote peer 6dff54dc0873484fa326b8b7d553eb5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:50:17.450160 15675 consensus_queue.cc:1035] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:50:17.453914 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.453310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:60395)
0929 14:50:17.453598 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.453710 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260929 14:50:17.457870 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.457243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:45783)
0929 14:50:17.457617 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.457729 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
I20260929 14:50:17.458165 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.457894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:36673)
0929 14:50:17.458030 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.458087 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:50:17.808121 15683 ts_tablet_manager.cc:927] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Initiating tablet copy from peer 6dff54dc0873484fa326b8b7d553eb5c (127.0.98.1:41319)
I20260929 14:50:17.809098 15683 tablet_copy_client.cc:316] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:41319
Sep 29 14:50:17 dist-test-slave-5hd1 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:50:17.822477 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.809884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41319 (local address 127.0.98.3:51585)
0929 14:50:17.810313 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.810352 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:17.810572 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:17.811125 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:17.811131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:17.811612 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:17.811780 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:17.811786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:17.812863 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:17.812866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:17.813644 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:17.813655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:17.814280 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:17.814282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:17.814398 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:17.815135 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:17.815149 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:17.817064 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:17.819511 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:17.819525 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:17.819530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:17.819906 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:17.820396 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:17.820400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:17.820402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:17.820567 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:17.821535 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:17.821545 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:17.821720 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 14:50:17.821911 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 14:50:17.822239 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":134,"threads_started":1}
I20260929 14:50:17.822476 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.810077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51585 (local address 127.0.98.1:41319)
0929 14:50:17.810505 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:17.810523 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:17.810573 (+ 50us) server_negotiation.cc:407] Connection header received
0929 14:50:17.810726 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:17.810732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:17.810940 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:17.811080 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:17.811914 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:17.812736 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:17.813767 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:17.814133 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:17.814223 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:17.817240 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:17.817249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:17.817262 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:17.817286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:17.819365 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:17.820062 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:17.820067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:17.820068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:17.820156 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:17.820774 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:17.820778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:17.820779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:17.821371 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:17.821567 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:17.822015 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 14:50:17.822271 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260929 14:50:17.824090 15367 tablet_copy_service.cc:140] P 6dff54dc0873484fa326b8b7d553eb5c: Received BeginTabletCopySession request for tablet 36093c5c4b404c479fef50950ab879ac from peer 40bd93703c39466890e845494fa4bd4a ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51585)
I20260929 14:50:17.824432 15367 tablet_copy_service.cc:161] P 6dff54dc0873484fa326b8b7d553eb5c: Beginning new tablet copy session on tablet 36093c5c4b404c479fef50950ab879ac from peer 40bd93703c39466890e845494fa4bd4a at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51585: session id = 40bd93703c39466890e845494fa4bd4a-36093c5c4b404c479fef50950ab879ac
I20260929 14:50:17.826007 15367 tablet_copy_source_session.cc:205] T 36093c5c4b404c479fef50950ab879ac P 6dff54dc0873484fa326b8b7d553eb5c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:50:17.831246 15683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36093c5c4b404c479fef50950ab879ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:17.838158 15683 tablet_copy_client.cc:791] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: tablet copy: Starting download of 0 data blocks...
I20260929 14:50:17.838425 15683 tablet_copy_client.cc:655] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: tablet copy: Starting download of 1 WAL segments...
I20260929 14:50:17.841002 15683 tablet_copy_client.cc:531] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:50:17.842885 15683 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Bootstrap starting.
I20260929 14:50:17.848840 15683 log.cc:824] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:17.856652 15683 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:17.857021 15683 tablet_bootstrap.cc:492] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Bootstrap complete.
I20260929 14:50:17.857379 15683 ts_tablet_manager.cc:1397] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260929 14:50:17.859750 15683 raft_consensus.cc:348] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:17.859990 15683 raft_consensus.cc:729] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bd93703c39466890e845494fa4bd4a, State: Initialized, Role: FOLLOWER
I20260929 14:50:17.860467 15683 consensus_queue.cc:260] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40bd93703c39466890e845494fa4bd4a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46193 } } peers { permanent_uuid: "bd67a48869314277bf81865d86b6bc5d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41119 } } peers { permanent_uuid: "6dff54dc0873484fa326b8b7d553eb5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41319 } }
I20260929 14:50:17.861402 15683 ts_tablet_manager.cc:1428] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260929 14:50:17.862149 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.861841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:34773)
0929 14:50:17.861987 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.862055 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:50:17.862485 15367 tablet_copy_service.cc:342] P 6dff54dc0873484fa326b8b7d553eb5c: Request end of tablet copy session 40bd93703c39466890e845494fa4bd4a-36093c5c4b404c479fef50950ab879ac received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51585
I20260929 14:50:17.862628 15367 tablet_copy_service.cc:433] P 6dff54dc0873484fa326b8b7d553eb5c: ending tablet copy session 40bd93703c39466890e845494fa4bd4a-36093c5c4b404c479fef50950ab879ac on tablet 36093c5c4b404c479fef50950ab879ac with peer 40bd93703c39466890e845494fa4bd4a
I20260929 14:50:17.863111 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:17.862855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:57839)
0929 14:50:17.862984 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:17.863051 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 14:50:18.386595 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:18.386159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:55721)
0929 14:50:18.386419 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:18.386480 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260929 14:50:18.386633 15616 raft_consensus.cc:1203] T 36093c5c4b404c479fef50950ab879ac P 40bd93703c39466890e845494fa4bd4a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 14:50:18.388401 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:18.387864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:39963)
0929 14:50:18.388152 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:18.388297 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260929 14:50:18.459681 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:18.459199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:38721)
0929 14:50:18.459485 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:18.459561 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260929 14:50:19.388481 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:19.387857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:53795)
0929 14:50:19.388305 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:19.388378 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260929 14:50:19.389779 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:19.389193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:34109)
0929 14:50:19.389543 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:19.389651 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
I20260929 14:50:19.461159 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:19.460603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:54815)
0929 14:50:19.460952 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:19.461032 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":76,"threads_started":1}
I20260929 14:50:20.390218 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:20.389630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:46273)
0929 14:50:20.389991 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:20.390083 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":89,"threads_started":1}
I20260929 14:50:20.390992 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:20.390572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:60167)
0929 14:50:20.390835 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:20.390889 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260929 14:50:20.462657 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:20.462084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:48745)
0929 14:50:20.462442 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:20.462530 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260929 14:50:21.392128 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:21.391540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:40379)
0929 14:50:21.391907 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:21.391984 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
I20260929 14:50:21.392529 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:21.391968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:58565)
0929 14:50:21.392322 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:21.392398 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":85,"threads_started":1}
I20260929 14:50:21.392946 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:21.392687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:57761)
0929 14:50:21.392824 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:21.392882 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:50:21.464304 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:21.463752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:35237)
0929 14:50:21.464062 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:21.464133 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260929 14:50:22.394070 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:22.393512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:33243)
0929 14:50:22.393818 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:22.393928 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260929 14:50:22.394363 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:22.393803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:55449)
0929 14:50:22.394154 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:22.394229 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260929 14:50:22.465835 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:22.465229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:45787)
0929 14:50:22.465647 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:22.465720 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260929 14:50:23.395665 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:23.395073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:46751)
0929 14:50:23.395418 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:23.395506 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":77,"threads_started":1}
I20260929 14:50:23.395736 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:23.395287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:52711)
0929 14:50:23.395581 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:23.395640 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260929 14:50:23.467465 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:23.466938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:60869)
0929 14:50:23.467254 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:23.467335 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
I20260929 14:50:24.397610 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:24.396842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:59733)
0929 14:50:24.397309 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:24.397420 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260929 14:50:24.397680 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:24.396950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:52201)
0929 14:50:24.397399 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:24.397494 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":114,"threads_started":1}
I20260929 14:50:24.468947 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:24.468413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:46595)
0929 14:50:24.468731 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:24.468818 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
I20260929 14:50:25.399150 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:25.398679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:51773)
0929 14:50:25.398982 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:25.399046 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260929 14:50:25.399300 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:25.398616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:59303)
0929 14:50:25.399086 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:25.399154 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":155,"threads_started":1}
I20260929 14:50:25.470547 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:25.469988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:37911)
0929 14:50:25.470360 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:25.470435 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":149,"threads_started":1}
I20260929 14:50:26.400774 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:26.400332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:59967)
0929 14:50:26.400627 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:26.400681 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":128,"threads_started":1}
I20260929 14:50:26.400906 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:26.400397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:50287)
0929 14:50:26.400689 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:26.400748 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260929 14:50:26.471931 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:26.471420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:54183)
0929 14:50:26.471731 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:26.471815 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260929 14:50:27.402320 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:27.401663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:44331)
0929 14:50:27.402015 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:27.402115 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260929 14:50:27.402567 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:27.401908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:50581)
0929 14:50:27.402310 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:27.402429 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":102,"threads_started":1}
I20260929 14:50:27.473491 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:27.472938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:40929)
0929 14:50:27.473274 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:27.473385 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260929 14:50:28.403839 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:28.403383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:54609)
0929 14:50:28.403657 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:28.403722 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
I20260929 14:50:28.403841 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:28.403201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:33191)
0929 14:50:28.403645 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:28.403731 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":141,"threads_started":1}
I20260929 14:50:28.475099 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:28.474464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:52779)
0929 14:50:28.474906 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:28.474996 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
I20260929 14:50:29.405350 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:29.404844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:51441)
0929 14:50:29.405177 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:29.405252 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260929 14:50:29.405514 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:29.404827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:52487)
0929 14:50:29.405257 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:29.405353 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260929 14:50:29.476590 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:29.476076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:35755)
0929 14:50:29.476387 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:29.476458 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":135,"threads_started":1}
I20260929 14:50:30.406920 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:30.406400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.3:44913)
0929 14:50:30.406703 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:30.406792 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
I20260929 14:50:30.407107 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:30.406522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.1:59677)
0929 14:50:30.406899 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:30.406958 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260929 14:50:30.478010 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:30.477466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37801 (local address 127.0.98.2:42963)
0929 14:50:30.477812 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:30.477884 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260929 14:50:30.966500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15229
I20260929 14:50:30.973124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15397
I20260929 14:50:30.980167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15532
2026-09-29T14:50:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19567 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:50:31 dist-test-slave-5hd1 krb5kdc[15734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:50:31 dist-test-slave-5hd1 krb5kdc[15734](info): set up 2 sockets
Sep 29 14:50:31 dist-test-slave-5hd1 krb5kdc[15734](info): commencing operation
krb5kdc: starting...
W20260929 14:50:32.036839 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:50:32.175361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:32.295763 15747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:32.296114 15747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:32.296224 15747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:32.300240 15747 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:50:32.300334 15747 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:50:32.300365 15747 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:50:32.300405 15747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:32.300451 15747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:32.300496 15747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:32.300530 15747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:32.300570 15747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:32.305970 15747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:32.307142 15747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:32.308254 15747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:32.318015 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:32.318002 15755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:32.318037 15747 server_base.cc:1034] running on GCE node
W20260929 14:50:32.318018 15752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:32.319079 15747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:32.320626 15747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:32.321885 15747 hybrid_clock.cc:648] HybridClock initialized: now 1790693432321849 us; error 58 us; skew 500 ppm
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:32.325445 15747 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:32.326619 15747 webserver.cc:460] Webserver started at http://127.0.98.62:43331/ using document root <none> and password file <none>
I20260929 14:50:32.327318 15747 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:32.327417 15747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:32.327697 15747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:32.330313 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "9e8bf2f9df4e48aba347d9f52c182ef1"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "8aa54862591f3923a512a632d5a33536"
server_key_iv: "346423046e42bcea2946d9bc73b0a542"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.330993 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e8bf2f9df4e48aba347d9f52c182ef1"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "8aa54862591f3923a512a632d5a33536"
server_key_iv: "346423046e42bcea2946d9bc73b0a542"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.335740 15747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 14:50:32.338863 15762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.340389 15747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:32.340543 15747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "9e8bf2f9df4e48aba347d9f52c182ef1"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "8aa54862591f3923a512a632d5a33536"
server_key_iv: "346423046e42bcea2946d9bc73b0a542"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.340699 15747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:32.367559 15747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:32.371641 15747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:32.371928 15747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:32.381240 15747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36921
I20260929 14:50:32.381582 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36921 every 8 connection(s)
I20260929 14:50:32.382490 15747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:32.383805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15747
I20260929 14:50:32.383926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:50:32.384366 392 external_mini_cluster.cc:1428] Setting key a08f6248733513098f388c18ff891f1c
I20260929 14:50:32.385960 15815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:32.392694 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap starting.
I20260929 14:50:32.394979 15815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Neither blocks nor log segments found. Creating new log.
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:32.395859 15815 log.cc:824] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:32.397743 15815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: No bootstrap required, opened a new log
I20260929 14:50:32.401549 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:32.386162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.0.98.62:36921)
0929 14:50:32.386691 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:32.386712 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:32.386763 (+ 51us) server_negotiation.cc:407] Connection header received
0929 14:50:32.387683 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:32.387705 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:32.388125 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:32.388542 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:32.389422 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.390895 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.391849 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.392541 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.392717 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:32.395876 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:32.395884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:32.395904 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:32.395970 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:32.398099 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.398659 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.398664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.398669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.398790 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.399097 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.399099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.399100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.399599 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:32.399760 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:32.400231 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 14:50:32.400632 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":149,"threads_started":1}
I20260929 14:50:32.401763 15815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:32.402076 15815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:32.402170 15815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Initialized, Role: FOLLOWER
I20260929 14:50:32.402601 15815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:32.402753 15815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:32.402878 15815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:32.402987 15815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:32.404103 15815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:32.404505 15815 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8bf2f9df4e48aba347d9f52c182ef1; no voters:
I20260929 14:50:32.404771 15815 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:50:32.404948 15820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:32.405254 15820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Running, Role: LEADER
I20260929 14:50:32.405701 15820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:32.406073 15815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:32.407680 15821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:50:32.407845 15821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:32.407725 15822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8bf2f9df4e48aba347d9f52c182ef1. Latest consensus state: current_term: 1 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:50:32.408133 15822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
W20260929 14:50:32.411339 15835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:50:32.411423 15835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:50:32.411513 15833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:32.412298 15833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:32.418622 15833 catalog_manager.cc:1347] Generated new cluster ID: cbf09167d204421eb3aa1456f3115d43
I20260929 14:50:32.418696 15833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:32.428783 15833 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:50:32.430079 15833 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:32.434558 15833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Generated new TSK 0
I20260929 14:50:32.435736 15833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:50:32.472517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:32.592232 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:32.592568 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:32.592643 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:32.596915 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:32.597010 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:32.597105 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:32.602530 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:32.603672 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:32.604775 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:32.615716 15846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:32.615731 15847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:32.615769 15849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:32.615756 15841 server_base.cc:1034] running on GCE node
I20260929 14:50:32.616611 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:32.617501 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:32.618731 15841 hybrid_clock.cc:648] HybridClock initialized: now 1790693432618680 us; error 52 us; skew 500 ppm
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:32.622202 15841 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:32.623317 15841 webserver.cc:460] Webserver started at http://127.0.98.1:42793/ using document root <none> and password file <none>
I20260929 14:50:32.624035 15841 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:32.624133 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:32.624431 15841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:32.626883 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "00f617bde105bcd5ab3a6ec70e47416a"
server_key_iv: "04ef1a8ab271a1a64c148258191d9970"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.627538 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "00f617bde105bcd5ab3a6ec70e47416a"
server_key_iv: "04ef1a8ab271a1a64c148258191d9970"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.631986 15841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260929 14:50:32.635089 15856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.636901 15841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:32.637094 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "00f617bde105bcd5ab3a6ec70e47416a"
server_key_iv: "04ef1a8ab271a1a64c148258191d9970"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.637252 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:32.661713 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:32.666045 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:32.666394 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:32.667289 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:32.668565 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:32.668653 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.668738 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:32.668782 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.680856 15841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39543
I20260929 14:50:32.680917 15969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39543 every 8 connection(s)
I20260929 14:50:32.681912 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:32.691584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15841
I20260929 14:50:32.691704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:50:32.692054 392 external_mini_cluster.cc:1428] Setting key 2adc3d97cb2f96ff811044ed246d6b40
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:32.700300 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:32.684565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46375 (local address 127.0.98.62:36921)
0929 14:50:32.684794 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:32.684799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:32.685894 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 14:50:32.687586 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:32.687590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:32.687674 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:32.687764 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:32.689498 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.690122 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.691469 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.691825 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.691962 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:32.695455 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:32.695462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:32.695469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:32.695516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:32.697492 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.698417 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.698421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.698424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.698534 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.698931 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.698934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.698936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.699171 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:32.699286 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:32.699900 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 14:50:32.700077 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 14:50:32.701373 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:32.684957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:46375)
0929 14:50:32.685547 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:32.685613 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:32.687304 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:32.687990 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:32.687998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:32.688559 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:32.689300 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:32.689318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.690306 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.690309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:32.691326 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:32.691337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.692026 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.692030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:32.692325 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:32.693049 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:32.693075 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:32.695159 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:32.697691 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:32.697698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:32.697718 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:32.698137 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:32.698642 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:32.698646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:32.698648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:32.698816 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:32.699295 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:32.699305 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:32.699633 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 14:50:32.699860 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:50:32.700320 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:50:32.703083 15970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:32.703429 15970 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:32.704041 15970 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:50:32.705941 15779 ts_manager.cc:194] Registered new tserver with Master: 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543)
I20260929 14:50:32.707542 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:50:32.725548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:32.865701 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:32.866074 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:32.866127 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:32.870810 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:32.870916 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:32.870988 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:32.877619 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:32.879125 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:32.880520 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:32.892640 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:32.892640 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:32.892647 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:32.892829 15975 server_base.cc:1034] running on GCE node
I20260929 14:50:32.893642 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:32.894412 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:32.895632 15975 hybrid_clock.cc:648] HybridClock initialized: now 1790693432895616 us; error 39 us; skew 500 ppm
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:32.898964 15975 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:32.900014 15975 webserver.cc:460] Webserver started at http://127.0.98.2:40895/ using document root <none> and password file <none>
I20260929 14:50:32.900744 15975 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:32.900851 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:32.901103 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:32.903234 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ef70f7de9db431b804604a43dfef81f"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "ff3d3bab11dc7212d70df4c7bcb5b055"
server_key_iv: "59a45449e671bc3dc810864a606cf25b"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.903891 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef70f7de9db431b804604a43dfef81f"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "ff3d3bab11dc7212d70df4c7bcb5b055"
server_key_iv: "59a45449e671bc3dc810864a606cf25b"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.908474 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:50:32.911306 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.912825 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:32.912992 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "9ef70f7de9db431b804604a43dfef81f"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "ff3d3bab11dc7212d70df4c7bcb5b055"
server_key_iv: "59a45449e671bc3dc810864a606cf25b"
server_key_version: "encryptionkey@0"
I20260929 14:50:32.913108 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:32.929992 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:32.933511 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:32.933766 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:32.934481 15975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:32.935336 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:32.935389 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.935428 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:32.935443 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:32.946005 15975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36165
I20260929 14:50:32.946080 16103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36165 every 8 connection(s)
I20260929 14:50:32.947018 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:32.954669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 15975
I20260929 14:50:32.954814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:50:32.955111 392 external_mini_cluster.cc:1428] Setting key d51711813bf65838fd27deed969f9a7f
Sep 29 14:50:32 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:32.963884 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:32.949486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58159 (local address 127.0.98.62:36921)
0929 14:50:32.949688 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:32.949694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:32.950839 (+ 1145us) server_negotiation.cc:407] Connection header received
0929 14:50:32.952263 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:32.952267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:32.952330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:32.952430 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:32.953774 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.954353 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.955998 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.956347 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.956461 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:32.959684 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:32.959692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:32.959697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:32.959738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:32.961494 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.962116 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.962121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.962123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.962221 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:32.962604 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:32.962608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:32.962610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:32.962834 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:32.962939 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:32.963541 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 14:50:32.963695 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:50:32.965112 16106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:32.950023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:58159)
0929 14:50:32.950649 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:32.950701 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:32.951986 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:32.952644 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:32.952651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:32.953057 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:32.953590 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:32.953603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.954511 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.954516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:32.955859 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:32.955870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:32.956510 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:32.956513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:32.956694 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:32.957505 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:32.957527 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:32.959465 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:32.961641 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:32.961649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:32.961671 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:32.962000 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:32.962324 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:32.962328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:32.962330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:32.962493 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:32.962952 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:32.962962 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:32.963275 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 14:50:32.963500 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 14:50:32.963934 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":170,"threads_started":1}
I20260929 14:50:32.966920 16104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:32.967270 16104 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:32.967996 16104 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:50:32.969645 15779 ts_manager.cc:194] Registered new tserver with Master: 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
I20260929 14:50:32.970605 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:50:32.988806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:33.112110 16109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:33.112465 16109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:33.112511 16109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:33.116667 16109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:33.116726 16109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:33.116787 16109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:33.122179 16109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:33.123222 16109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:33.124327 16109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:33.134722 16114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:33.134897 16109 server_base.cc:1034] running on GCE node
W20260929 14:50:33.134722 16117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:33.134793 16115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:33.135536 16109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:33.136267 16109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:33.137459 16109 hybrid_clock.cc:648] HybridClock initialized: now 1790693433137435 us; error 46 us; skew 500 ppm
Sep 29 14:50:33 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:33.140594 16109 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:33.141631 16109 webserver.cc:460] Webserver started at http://127.0.98.3:38363/ using document root <none> and password file <none>
I20260929 14:50:33.142263 16109 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:33.142370 16109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:33.142592 16109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:50:33.144706 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "5426692ebc73469489d7f231b2846cf1"
format_stamp: "Formatted at 2026-09-29 14:50:33 on dist-test-slave-5hd1"
server_key: "454113ac6edc0d8b015495da25d72a2e"
server_key_iv: "17ee9290354a345246cc86050e189a9b"
server_key_version: "encryptionkey@0"
I20260929 14:50:33.145326 16109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5426692ebc73469489d7f231b2846cf1"
format_stamp: "Formatted at 2026-09-29 14:50:33 on dist-test-slave-5hd1"
server_key: "454113ac6edc0d8b015495da25d72a2e"
server_key_iv: "17ee9290354a345246cc86050e189a9b"
server_key_version: "encryptionkey@0"
I20260929 14:50:33.149459 16109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:50:33.152289 16124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:33.153743 16109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:33.153906 16109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "5426692ebc73469489d7f231b2846cf1"
format_stamp: "Formatted at 2026-09-29 14:50:33 on dist-test-slave-5hd1"
server_key: "454113ac6edc0d8b015495da25d72a2e"
server_key_iv: "17ee9290354a345246cc86050e189a9b"
server_key_version: "encryptionkey@0"
I20260929 14:50:33.154042 16109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:33.173055 16109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:33.176630 16109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:33.176831 16109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:33.177439 16109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:33.178267 16109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:33.178339 16109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:33.178413 16109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:33.178468 16109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:33.188782 16109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33717
I20260929 14:50:33.188849 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33717 every 8 connection(s)
I20260929 14:50:33.189723 16109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:33.197877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16109
I20260929 14:50:33.198066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:50:33.198457 392 external_mini_cluster.cc:1428] Setting key 6f6b398644f627a12b7ebff00ffd0004
Sep 29 14:50:33 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:33.207783 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.192226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39665 (local address 127.0.98.62:36921)
0929 14:50:33.192431 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.192436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.193381 (+ 945us) server_negotiation.cc:407] Connection header received
0929 14:50:33.194879 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.194884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.194941 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.195042 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:33.196609 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.197188 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.199190 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.199501 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.199621 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.203409 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:33.203412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:33.203414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:33.203449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:33.205134 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:33.205761 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:33.205764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:33.205766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:33.205848 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:33.206327 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:33.206332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:33.206333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:33.206572 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:33.206705 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.207423 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.207581 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 14:50:33.208971 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.192590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:39665)
0929 14:50:33.193139 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.193205 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.194613 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.195267 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.195275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.195752 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:33.196440 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.196457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.197348 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.197351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.199004 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.199014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.199711 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.199725 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.199988 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.200946 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:33.200990 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:33.203195 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:33.205270 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:33.205276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:33.205293 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:33.205606 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:33.205957 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:33.205959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:33.205961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:33.206115 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:33.206700 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:33.206708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:33.207069 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.207319 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.207747 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260929 14:50:33.210604 16238 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:33.210888 16238 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:33.211522 16238 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
I20260929 14:50:33.212926 15779 ts_manager.cc:194] Registered new tserver with Master: 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717)
I20260929 14:50:33.213639 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:33.213809 15779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39665
I20260929 14:50:33.224575 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.216116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.98.62:36921)
0929 14:50:33.216285 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.216289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.216576 (+ 287us) server_negotiation.cc:407] Connection header received
0929 14:50:33.216779 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.216781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.216825 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.216910 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:33.217713 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.218289 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.219164 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.219418 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.219532 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.221118 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:33.221123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:33.221126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:33.221154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:33.222723 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:33.223162 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:33.223167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:33.223169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:33.223265 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:33.223546 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:33.223551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:33.223553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:33.223821 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:33.223978 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.224255 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.224388 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 14:50:33.228255 15779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:50:33.232775 15779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:50:33.250669 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.244044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33717 (local address 127.0.0.1:54846)
0929 14:50:33.244617 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.244674 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.244883 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.245773 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.245779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.245804 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:33.245975 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.245982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.247398 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.247401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.248836 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.248846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.249801 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.249805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.249933 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.250006 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.250129 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.250478 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":180,"threads_started":1}
I20260929 14:50:33.251344 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.245241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.0.98.2:36165)
0929 14:50:33.245679 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.245690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.245712 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:50:33.245782 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.245787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.245944 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.246342 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:33.246503 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.247776 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.249441 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.250189 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.250984 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.251118 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.251163 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.251235 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":174,"threads_started":1}
I20260929 14:50:33.251355 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.244230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.0.98.3:33717)
0929 14:50:33.244629 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.244641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.244823 (+ 182us) server_negotiation.cc:407] Connection header received
0929 14:50:33.245081 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.245089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.245402 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.245571 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:33.246103 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.247249 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.248973 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.249660 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.249770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.249926 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.250981 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.251134 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":185,"threads_started":1}
I20260929 14:50:33.250903 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.244983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.0.1:58436)
0929 14:50:33.245278 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.245296 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.245403 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.246103 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.246106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.246124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:33.246210 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.246215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.247931 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.247935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.249287 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.249297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.250325 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.250328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.250437 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.250467 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.250521 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.250630 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260929 14:50:33.251223 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.245350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39543 (local address 127.0.0.1:49592)
0929 14:50:33.245786 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.245801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.245903 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.246613 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.246617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.246640 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:33.246795 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.246803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.248378 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.248381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.249486 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.249492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.250833 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.250837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.250945 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.250977 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.251025 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.251102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":93,"threads_started":1}
I20260929 14:50:33.251852 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.245814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.0.98.1:39543)
0929 14:50:33.246152 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.246158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.246180 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:50:33.246253 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.246257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.246454 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.246948 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:33.247086 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.248092 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.249604 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.250653 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.251332 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.251515 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.251568 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.251699 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":132,"threads_started":1}
I20260929 14:50:33.254598 16038 tablet_service.cc:1467] Processing CreateTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 (DEFAULT_TABLE table=test-table [id=e9f680e49812497190fc8b91faaa7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:33.255661 16038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421976c587ea4738bd0ede5cb2c23d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:33.255218 15904 tablet_service.cc:1467] Processing CreateTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 (DEFAULT_TABLE table=test-table [id=e9f680e49812497190fc8b91faaa7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:33.255224 16171 tablet_service.cc:1467] Processing CreateTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 (DEFAULT_TABLE table=test-table [id=e9f680e49812497190fc8b91faaa7f64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:33.256448 15904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421976c587ea4738bd0ede5cb2c23d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:33.256453 16171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421976c587ea4738bd0ede5cb2c23d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:33.263183 16255 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Bootstrap starting.
I20260929 14:50:33.265056 16256 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap starting.
I20260929 14:50:33.265556 16257 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Bootstrap starting.
I20260929 14:50:33.266062 16255 tablet_bootstrap.cc:654] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:33.266973 16255 log.cc:824] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:33.267769 16257 tablet_bootstrap.cc:654] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:33.268476 16256 tablet_bootstrap.cc:654] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:33.269130 16257 log.cc:824] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:33.269443 16256 log.cc:824] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:33.269999 16255 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: No bootstrap required, opened a new log
I20260929 14:50:33.270287 16255 ts_tablet_manager.cc:1397] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260929 14:50:33.271171 16257 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: No bootstrap required, opened a new log
I20260929 14:50:33.271404 16257 ts_tablet_manager.cc:1397] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260929 14:50:33.271865 16256 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: No bootstrap required, opened a new log
I20260929 14:50:33.272097 16256 ts_tablet_manager.cc:1397] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:50:33.274525 16257 raft_consensus.cc:348] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.274801 16257 raft_consensus.cc:374] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:33.274860 16257 raft_consensus.cc:729] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5426692ebc73469489d7f231b2846cf1, State: Initialized, Role: FOLLOWER
I20260929 14:50:33.275315 16257 consensus_queue.cc:260] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.275141 16255 raft_consensus.cc:348] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.275477 16255 raft_consensus.cc:374] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:33.275542 16255 raft_consensus.cc:729] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef70f7de9db431b804604a43dfef81f, State: Initialized, Role: FOLLOWER
I20260929 14:50:33.276046 16255 consensus_queue.cc:260] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.276535 16257 ts_tablet_manager.cc:1428] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260929 14:50:33.276893 16238 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
I20260929 14:50:33.277233 16255 ts_tablet_manager.cc:1428] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 14:50:33.277284 16104 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
I20260929 14:50:33.277171 16256 raft_consensus.cc:348] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.277456 16256 raft_consensus.cc:374] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:33.277516 16256 raft_consensus.cc:729] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633f9e7d0b9d4f738f3eb09163b94a82, State: Initialized, Role: FOLLOWER
I20260929 14:50:33.277988 16256 consensus_queue.cc:260] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.279153 16256 ts_tablet_manager.cc:1428] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:50:33.279196 15970 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
I20260929 14:50:33.314464 16263 raft_consensus.cc:482] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:33.314785 16263 raft_consensus.cc:504] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.316162 16263 leader_election.cc:288] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
I20260929 14:50:33.323203 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.316753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42321 (local address 127.0.98.3:33717)
0929 14:50:33.316980 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.316986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.317274 (+ 288us) server_negotiation.cc:407] Connection header received
0929 14:50:33.317653 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.317657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.317763 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.317872 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:33.318334 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.319184 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.321501 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.322221 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.322355 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.322437 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.322938 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.323031 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 14:50:33.323210 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.317380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53851 (local address 127.0.98.2:36165)
0929 14:50:33.317575 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.317582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.317703 (+ 121us) server_negotiation.cc:407] Connection header received
0929 14:50:33.317839 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.317844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.317902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.317981 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:33.318376 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.319187 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.321501 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.322247 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.322355 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.322437 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.322938 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.323031 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 14:50:33.323292 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.317217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.98.1:53851)
0929 14:50:33.317581 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.317599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.317713 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.318070 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.318073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.318088 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:33.318243 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.318252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.319337 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.319341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.321319 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.321332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.322426 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.322431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.322727 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.322779 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.322883 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.323035 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":374,"thread_start_us":120,"threads_started":1}
I20260929 14:50:33.323302 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.316652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33717 (local address 127.0.98.1:42321)
0929 14:50:33.317138 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.317155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:33.317408 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:33.317911 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:33.317914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:33.317934 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:33.318165 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.318173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.319346 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.319349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.321320 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:33.321332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.322426 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.322431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:33.322721 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.322777 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 14:50:33.322882 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 14:50:33.323035 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":484,"thread_start_us":182,"threads_started":1}
I20260929 14:50:33.324239 16058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421976c587ea4738bd0ede5cb2c23d19" candidate_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef70f7de9db431b804604a43dfef81f" is_pre_election: true
I20260929 14:50:33.324381 16192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421976c587ea4738bd0ede5cb2c23d19" candidate_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426692ebc73469489d7f231b2846cf1" is_pre_election: true
I20260929 14:50:33.324636 16058 raft_consensus.cc:2454] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633f9e7d0b9d4f738f3eb09163b94a82 in term 0.
I20260929 14:50:33.324651 16192 raft_consensus.cc:2454] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633f9e7d0b9d4f738f3eb09163b94a82 in term 0.
I20260929 14:50:33.325395 15859 leader_election.cc:302] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82, 9ef70f7de9db431b804604a43dfef81f; no voters:
I20260929 14:50:33.325829 16263 raft_consensus.cc:2781] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:33.325925 16263 raft_consensus.cc:482] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:33.325960 16263 raft_consensus.cc:3037] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:33.327214 16263 raft_consensus.cc:504] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.327626 16263 leader_election.cc:288] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 1 election: Requested vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
I20260929 14:50:33.328137 16192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421976c587ea4738bd0ede5cb2c23d19" candidate_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426692ebc73469489d7f231b2846cf1"
I20260929 14:50:33.328255 16058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421976c587ea4738bd0ede5cb2c23d19" candidate_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef70f7de9db431b804604a43dfef81f"
I20260929 14:50:33.328310 16192 raft_consensus.cc:3037] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:33.328440 16058 raft_consensus.cc:3037] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:33.329769 16058 raft_consensus.cc:2454] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633f9e7d0b9d4f738f3eb09163b94a82 in term 1.
I20260929 14:50:33.329769 16192 raft_consensus.cc:2454] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633f9e7d0b9d4f738f3eb09163b94a82 in term 1.
I20260929 14:50:33.330152 15859 leader_election.cc:302] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5426692ebc73469489d7f231b2846cf1, 633f9e7d0b9d4f738f3eb09163b94a82; no voters:
I20260929 14:50:33.330454 16263 raft_consensus.cc:2781] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:33.330772 16263 raft_consensus.cc:686] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 LEADER]: Becoming Leader. State: Replica: 633f9e7d0b9d4f738f3eb09163b94a82, State: Running, Role: LEADER
I20260929 14:50:33.331213 16263 consensus_queue.cc:237] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:33.335222 15777 catalog_manager.cc:5495] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } health_report { overall_health: HEALTHY } } }
I20260929 14:50:33.355903 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.351524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.0.98.1:39543)
0929 14:50:33.351699 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:33.351707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:33.351726 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:50:33.351842 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:33.351845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:33.351917 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:33.352052 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:33.352411 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.353060 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.354101 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:33.354329 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:33.354432 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:33.354804 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:33.355302 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:33.355430 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:33.355605 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 14:50:33.355719 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 14:50:33.364349 16058 raft_consensus.cc:1261] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Refusing update from remote peer 633f9e7d0b9d4f738f3eb09163b94a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:33.364349 16192 raft_consensus.cc:1261] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Refusing update from remote peer 633f9e7d0b9d4f738f3eb09163b94a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:33.365243 16266 consensus_queue.cc:1035] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:33.365505 16263 consensus_queue.cc:1035] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:33.373667 16269 mvcc.cc:205] Tried to move back new op lower bound from 7334680303050264576 to 7334680302928154624. Current Snapshot: MvccSnapshot[applied={T|T < 7334680303050264576}]
I20260929 14:50:33.376804 16271 mvcc.cc:205] Tried to move back new op lower bound from 7334680303050264576 to 7334680302928154624. Current Snapshot: MvccSnapshot[applied={T|T < 7334680303050264576}]
I20260929 14:50:33.383836 15778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:33.384052 15778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:33.386723 15778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Sending DeleteTablet for 3 replicas of tablet 421976c587ea4738bd0ede5cb2c23d19
I20260929 14:50:33.387621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15841
I20260929 14:50:33.387730 16038 tablet_service.cc:1514] Processing DeleteTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58436
I20260929 14:50:33.389514 16279 tablet_replica.cc:330] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: stopping tablet replica
I20260929 14:50:33.389436 16171 tablet_service.cc:1514] Processing DeleteTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54846
I20260929 14:50:33.389987 16279 raft_consensus.cc:2229] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:33.390074 16280 tablet_replica.cc:330] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: stopping tablet replica
I20260929 14:50:33.390420 16280 raft_consensus.cc:2229] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:33.390439 16279 raft_consensus.cc:2258] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:33.390785 16280 raft_consensus.cc:2258] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:33.392557 16279 ts_tablet_manager.cc:1905] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:33.392609 16280 ts_tablet_manager.cc:1905] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:33.395769 16280 ts_tablet_manager.cc:1918] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:33.396385 16280 log.cc:1196] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/421976c587ea4738bd0ede5cb2c23d19
I20260929 14:50:33.396261 16279 ts_tablet_manager.cc:1918] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:33.396755 16280 ts_tablet_manager.cc:1939] T 421976c587ea4738bd0ede5cb2c23d19 P 5426692ebc73469489d7f231b2846cf1: Deleting consensus metadata
W20260929 14:50:33.396732 15765 connection.cc:535] client connection to 127.0.98.1:39543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 14:50:33.396883 15765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:50:33.397058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15975
I20260929 14:50:33.399104 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.398713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39543 (local address 127.0.0.1:49606)
0929 14:50:33.398897 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.398997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 14:50:33.399682 15765 catalog_manager.cc:4584] TS 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 421976c587ea4738bd0ede5cb2c23d19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39543: connect: Connection refused (error 111)
I20260929 14:50:33.399935 15765 catalog_manager.cc:4841] TS 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): tablet 421976c587ea4738bd0ede5cb2c23d19 (table test-table [id=e9f680e49812497190fc8b91faaa7f64]) successfully deleted
I20260929 14:50:33.403934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16109
I20260929 14:50:33.404840 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:33.404254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.0.1:58442)
0929 14:50:33.404643 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:33.404716 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 14:50:33.411186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 15747
I20260929 14:50:53.419132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--webserver_interface=127.0.98.62
--webserver_port=43331
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:50:53.552831 16282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:53.553293 16282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:53.553381 16282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:53.558619 16282 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:50:53.558774 16282 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:50:53.558805 16282 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:50:53.558831 16282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:50:53.558869 16282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:53.558934 16282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:53.558979 16282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:50:53.559013 16282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:50:53.566167 16282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:53.567775 16282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:53.569137 16282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:53.585960 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:53.586021 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:53.585999 16282 server_base.cc:1034] running on GCE node
W20260929 14:50:53.586016 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:53.587060 16282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:53.588428 16282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:53.589624 16282 hybrid_clock.cc:648] HybridClock initialized: now 1790693453589583 us; error 68 us; skew 500 ppm
Sep 29 14:50:53 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:53.592957 16282 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:50:53.593973 16282 webserver.cc:460] Webserver started at http://127.0.98.62:43331/ using document root <none> and password file <none>
I20260929 14:50:53.594648 16282 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:53.594764 16282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:53.599117 16282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:53.601907 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:53.603395 16282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:50:53.603550 16282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "9e8bf2f9df4e48aba347d9f52c182ef1"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "8aa54862591f3923a512a632d5a33536"
server_key_iv: "346423046e42bcea2946d9bc73b0a542"
server_key_version: "encryptionkey@0"
I20260929 14:50:53.604184 16282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:53.627879 16282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:53.631170 16282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:53.631425 16282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:53.639379 16282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36921
I20260929 14:50:53.639674 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36921 every 8 connection(s)
I20260929 14:50:53.640717 16282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:50:53.646322 16351 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:50:53.647939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16282
I20260929 14:50:53.648568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39543
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=42793
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:53.648900 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap starting.
I20260929 14:50:53.671213 16351 log.cc:824] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:53.682379 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:53.682904 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap complete.
I20260929 14:50:53.687954 16351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:53.688613 16351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Initialized, Role: FOLLOWER
I20260929 14:50:53.689304 16351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:53.689486 16351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:50:53.689582 16351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:50:53.689834 16351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:50:53.692763 16351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:53.693230 16351 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8bf2f9df4e48aba347d9f52c182ef1; no voters:
I20260929 14:50:53.693670 16351 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:50:53.693784 16355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:50:53.694219 16355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 LEADER]: Becoming Leader. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Running, Role: LEADER
I20260929 14:50:53.694687 16355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:50:53.695312 16351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:50:53.696535 16357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8bf2f9df4e48aba347d9f52c182ef1. Latest consensus state: current_term: 2 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:50:53.696766 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:50:53.697044 16357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:53.697057 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:50:53.702072 16370 catalog_manager.cc:1259] Loaded cluster ID: cbf09167d204421eb3aa1456f3115d43
I20260929 14:50:53.702172 16370 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: loading cluster ID for follower catalog manager: success
I20260929 14:50:53.704385 16370 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: acquiring CA information for follower catalog manager: success
I20260929 14:50:53.705863 16370 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:50:53.706029 16364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:50:53.708949 16364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:50:53.709435 16364 catalog_manager.cc:1259] Loaded cluster ID: cbf09167d204421eb3aa1456f3115d43
I20260929 14:50:53.709501 16364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:50:53.710249 16364 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:50:53.710749 16364 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Loaded TSK: 0
I20260929 14:50:53.722339 16364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Generated new TSK 1
I20260929 14:50:53.722591 16364 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:50:53.775914 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:53.776259 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:53.776340 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:53.780413 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:53.780479 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:53.780539 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:50:53.785856 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42793
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:53.786972 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:53.788064 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:53.798482 16379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:53.798496 16378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:53.798828 16381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:53.798913 16353 server_base.cc:1034] running on GCE node
I20260929 14:50:53.799341 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:53.800050 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:53.801281 16353 hybrid_clock.cc:648] HybridClock initialized: now 1790693453801244 us; error 62 us; skew 500 ppm
Sep 29 14:50:53 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:53.804610 16353 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:50:53.805658 16353 webserver.cc:460] Webserver started at http://127.0.98.1:42793/ using document root <none> and password file <none>
I20260929 14:50:53.806248 16353 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:53.806368 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:53.810601 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:50:53.813191 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:53.814538 16353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:53.814713 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "00f617bde105bcd5ab3a6ec70e47416a"
server_key_iv: "04ef1a8ab271a1a64c148258191d9970"
server_key_version: "encryptionkey@0"
I20260929 14:50:53.815313 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:53.836864 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:53.840703 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:53.840955 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:53.841709 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:53.842988 16395 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:50:53.847504 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:50:53.847582 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260929 14:50:53.847635 16353 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:50:53.849491 16353 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:50:53.849560 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 14:50:53.849818 16395 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap starting.
I20260929 14:50:53.862635 16353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39543
I20260929 14:50:53.863080 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39543 every 8 connection(s)
I20260929 14:50:53.864156 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:50:53.867722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16353
I20260929 14:50:53.868356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36165
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=40895
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:50:53.873658 16395 log.cc:824] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:50:53 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:53.884673 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:53.867052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54125 (local address 127.0.98.62:36921)
0929 14:50:53.868164 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:53.868213 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:53.870063 (+ 1850us) server_negotiation.cc:407] Connection header received
0929 14:50:53.870361 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:53.870374 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:53.870749 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:53.871154 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:53.872514 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.873544 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.875133 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.875696 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.875879 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:53.878907 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:53.878913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:53.878927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:53.878957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:53.880925 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:53.881529 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:53.881535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:53.881538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:53.881660 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:53.882052 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:53.882056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:53.882057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:53.882454 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:53.882617 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:53.883276 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 14:50:53.883640 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":161,"threads_started":1}
I20260929 14:50:53.884732 16395 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:50:53.885203 16395 tablet_bootstrap.cc:492] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap complete.
I20260929 14:50:53.885434 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:53.866993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:54125)
0929 14:50:53.868423 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:53.868490 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:53.869851 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:53.871218 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:53.871227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:53.871729 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:53.872314 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:53.872335 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.873734 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.873737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:53.874999 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:53.875010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.875856 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.875860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:53.876064 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:53.876847 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:53.876866 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:53.878682 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:53.881079 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:53.881085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:53.881099 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:53.881383 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:53.881760 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:53.881765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:53.881766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:53.881923 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:53.882583 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:53.882590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:53.882876 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 14:50:53.884267 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0929 14:50:53.884623 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":156,"threads_started":1}
I20260929 14:50:53.886796 16395 ts_tablet_manager.cc:1397] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.004s
I20260929 14:50:53.887998 16503 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:53.888336 16503 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:53.888916 16503 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
I20260929 14:50:53.890141 16395 raft_consensus.cc:348] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:53.890743 16395 raft_consensus.cc:729] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633f9e7d0b9d4f738f3eb09163b94a82, State: Initialized, Role: FOLLOWER
I20260929 14:50:53.891278 16395 consensus_queue.cc:260] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:53.891752 16314 ts_manager.cc:194] Registered new tserver with Master: 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543)
I20260929 14:50:53.892392 16395 ts_tablet_manager.cc:1428] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 14:50:53.895563 16314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54125
I20260929 14:50:53.898516 16503 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
Sep 29 14:50:53 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:50:53.909183 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:53.895154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39543 (local address 127.0.0.1:54434)
0929 14:50:53.895856 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:53.895907 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:53.896227 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:53.897064 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:53.897071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:53.897588 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:53.897784 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:53.897793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.899104 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.899109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:53.900284 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:53.900298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.900879 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.900906 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:53.901032 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:53.901719 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:53.901736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:53.903794 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:53.906656 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:53.906666 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:53.906672 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:53.907039 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:53.907435 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:53.907439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:53.907441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:53.907551 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:53.908309 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:53.908316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:53.908483 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 14:50:53.908672 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 14:50:53.908947 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":139,"threads_started":1}
I20260929 14:50:53.909268 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:53.895192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.98.1:39543)
0929 14:50:53.896115 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:53.896130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:53.896165 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:50:53.896425 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:53.896431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:53.896657 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:53.896829 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:53.897952 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.898962 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.900422 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:53.900764 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:53.900855 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:53.903971 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:53.903980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:53.903994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:53.904022 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:53.906514 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:53.907210 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:53.907216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:53.907218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:53.907323 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:53.907735 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:53.907739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:53.907741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:53.908113 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:53.908332 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:53.908783 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 14:50:53.909036 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":152,"threads_started":1}
I20260929 14:50:53.911028 16437 tablet_service.cc:1514] Processing DeleteTablet for tablet 421976c587ea4738bd0ede5cb2c23d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54434
I20260929 14:50:53.912339 16512 tablet_replica.cc:330] stopping tablet replica
I20260929 14:50:53.912707 16512 raft_consensus.cc:2229] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:53.912990 16512 raft_consensus.cc:2258] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:53.914300 16512 ts_tablet_manager.cc:1905] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:53.919373 16512 ts_tablet_manager.cc:1918] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:53.919487 16512 log.cc:1196] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal/wals/421976c587ea4738bd0ede5cb2c23d19
I20260929 14:50:53.919800 16512 ts_tablet_manager.cc:1939] T 421976c587ea4738bd0ede5cb2c23d19 P 633f9e7d0b9d4f738f3eb09163b94a82: Deleting consensus metadata
I20260929 14:50:53.922158 16301 catalog_manager.cc:4841] TS 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543): tablet 421976c587ea4738bd0ede5cb2c23d19 (table test-table [id=e9f680e49812497190fc8b91faaa7f64]) successfully deleted
W20260929 14:50:54.000319 16507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:54.000586 16507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:54.000629 16507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:54.004726 16507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:54.004794 16507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:54.004853 16507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:50:54.010130 16507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40895
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:54.011160 16507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:54.012166 16507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:54.022734 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:54.022728 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:54.022769 16507 server_base.cc:1034] running on GCE node
W20260929 14:50:54.022727 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:54.023548 16507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:54.024295 16507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:54.025496 16507 hybrid_clock.cc:648] HybridClock initialized: now 1790693454025469 us; error 60 us; skew 500 ppm
Sep 29 14:50:54 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:54.028582 16507 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:50:54.029518 16507 webserver.cc:460] Webserver started at http://127.0.98.2:40895/ using document root <none> and password file <none>
I20260929 14:50:54.030079 16507 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:54.030202 16507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:54.034297 16507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:50:54.036823 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:54.038290 16507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:50:54.038470 16507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "9ef70f7de9db431b804604a43dfef81f"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "ff3d3bab11dc7212d70df4c7bcb5b055"
server_key_iv: "59a45449e671bc3dc810864a606cf25b"
server_key_version: "encryptionkey@0"
I20260929 14:50:54.039031 16507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:54.053363 16507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:54.056540 16507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:54.056749 16507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:54.057348 16507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:54.058485 16534 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 14:50:54.063347 16534 ts_tablet_manager.cc:1852] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 14:50:54.063827 16534 ts_tablet_manager.cc:1905] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:54.068907 16534 ts_tablet_manager.cc:1918] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:54.069042 16534 log.cc:1196] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/421976c587ea4738bd0ede5cb2c23d19
I20260929 14:50:54.069451 16534 ts_tablet_manager.cc:1939] T 421976c587ea4738bd0ede5cb2c23d19 P 9ef70f7de9db431b804604a43dfef81f: Deleting consensus metadata
I20260929 14:50:54.069850 16507 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 14:50:54.069955 16507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.000s
I20260929 14:50:54.070047 16507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:54.070101 16507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:54.081204 16507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36165
I20260929 14:50:54.081338 16641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36165 every 8 connection(s)
I20260929 14:50:54.082393 16507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:50:54.090122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16507
I20260929 14:50:54.090739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33717
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=38363
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:50:54 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:54.098683 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.084699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59243 (local address 127.0.98.62:36921)
0929 14:50:54.084911 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.084917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.086092 (+ 1175us) server_negotiation.cc:407] Connection header received
0929 14:50:54.087118 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.087121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.087181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.087284 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:54.088582 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.089185 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.090518 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.090920 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.091057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.094236 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:54.094240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:54.094242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:54.094274 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:54.096104 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.096841 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.096847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.096849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.096940 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.097316 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.097321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.097323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.097557 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:54.097718 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.098313 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.098488 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 14:50:54.099694 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.085266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:59243)
0929 14:50:54.085888 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.085939 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.086889 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.087477 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.087484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.087864 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:54.088390 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.088405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.089336 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.089338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.090360 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.090371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.091131 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.091135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.091318 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.092061 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:54.092078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:54.094012 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:54.096282 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:54.096289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:54.096311 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:54.096732 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:54.097044 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:54.097047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:54.097049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:54.097214 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:54.097710 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:54.097718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:54.098040 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.098262 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.098672 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":149,"threads_started":1}
I20260929 14:50:54.101344 16642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:54.101711 16642 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:54.102442 16642 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
I20260929 14:50:54.103781 16314 ts_manager.cc:194] Registered new tserver with Master: 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
I20260929 14:50:54.104564 16314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59243
W20260929 14:50:54.204049 16645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:50:54.204396 16645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:50:54.204493 16645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:50:54.208931 16645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:50:54.209007 16645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:50:54.209097 16645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:50:54.214720 16645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38363
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:50:54.215899 16645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:50:54.216998 16645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:50:54.227761 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:50:54.228116 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:54.228091 16645 server_base.cc:1034] running on GCE node
W20260929 14:50:54.227761 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:50:54.228873 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:50:54.229688 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:50:54.230916 16645 hybrid_clock.cc:648] HybridClock initialized: now 1790693454230893 us; error 39 us; skew 500 ppm
Sep 29 14:50:54 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:50:54.234344 16645 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:50:54.235517 16645 webserver.cc:460] Webserver started at http://127.0.98.3:38363/ using document root <none> and password file <none>
I20260929 14:50:54.236301 16645 fs_manager.cc:359] Metadata directory not provided
I20260929 14:50:54.236413 16645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:50:54.241329 16645 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:50:54.244059 16660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:54.245707 16645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:50:54.245880 16645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "5426692ebc73469489d7f231b2846cf1"
format_stamp: "Formatted at 2026-09-29 14:50:33 on dist-test-slave-5hd1"
server_key: "454113ac6edc0d8b015495da25d72a2e"
server_key_iv: "17ee9290354a345246cc86050e189a9b"
server_key_version: "encryptionkey@0"
I20260929 14:50:54.246574 16645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:50:54.271783 16645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:50:54.275859 16645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:50:54.276117 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:50:54.276908 16645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:50:54.277932 16645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:50:54.278005 16645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:54.278086 16645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:50:54.278129 16645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:50:54.290578 16645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33717
I20260929 14:50:54.290601 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33717 every 8 connection(s)
I20260929 14:50:54.291862 16645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:50:54.299295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16645
Sep 29 14:50:54 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:50:54.309295 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.293996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34717 (local address 127.0.98.62:36921)
0929 14:50:54.294181 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.294186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.295138 (+ 952us) server_negotiation.cc:407] Connection header received
0929 14:50:54.296805 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.296809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.296871 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.296959 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:54.298368 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.298999 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.300800 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.301108 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.301230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.304684 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:54.304687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:54.304689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:54.304715 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:54.306420 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.307149 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.307152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.307153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.307276 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.307784 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.307787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.307788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.308097 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:54.308271 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.308952 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.309099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:50:54.310348 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.294336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:34717)
0929 14:50:54.294882 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.294963 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.296571 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.297184 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.297195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.297646 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:50:54.298217 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.298230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.299211 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.299217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.300656 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.300670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.301288 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.301291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.301450 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.302283 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:50:54.302306 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:50:54.304451 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:50:54.306573 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:54.306580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:54.306602 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:54.307014 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:54.307385 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:50:54.307387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:50:54.307389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:50:54.307655 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:50:54.308294 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:50:54.308300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:50:54.308642 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.308886 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.309328 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":147,"threads_started":1}
I20260929 14:50:54.310614 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.300154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.0.98.62:36921)
0929 14:50:54.300524 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.300528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.300676 (+ 148us) server_negotiation.cc:407] Connection header received
0929 14:50:54.300908 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.300912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.300982 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.301077 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:54.302074 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.302925 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.304315 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.304737 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.304863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.306519 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:54.306524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:54.306527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:54.306561 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:54.308607 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.309038 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.309044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.309046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.309127 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.309638 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.309641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.309642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.309800 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:54.310023 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.310171 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.310301 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260929 14:50:54.311853 16774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36921
I20260929 14:50:54.312136 16774 heartbeater.cc:464] Registering TS with master...
I20260929 14:50:54.312853 16774 heartbeater.cc:511] Master 127.0.98.62:36921 requested a full tablet report, sending...
I20260929 14:50:54.314036 16315 ts_manager.cc:194] Registered new tserver with Master: 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717)
I20260929 14:50:54.314924 16315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34717
I20260929 14:50:54.324826 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:50:54.329483 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.325636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.0.98.62:36921)
0929 14:50:54.325860 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.325865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.326073 (+ 208us) server_negotiation.cc:407] Connection header received
0929 14:50:54.326345 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.326348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.326452 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.326568 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:54.326829 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.327365 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.328275 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.328503 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.328616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.328919 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:54.329058 (+ 139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 14:50:54.329335 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52044: token expired
Metrics: {"server-negotiator.queue_time_us":103}
W20260929 14:50:54.329555 16778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52044: token expired
W20260929 14:50:54.329948 16241 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36921)
I20260929 14:50:54.337369 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.330398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.0.98.62:36921)
0929 14:50:54.330541 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.330545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.330558 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:50:54.330648 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.330651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.330693 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.330788 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:50:54.331396 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.331865 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.332749 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.332955 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.333059 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.334468 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:50:54.334473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:50:54.334476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:50:54.334502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:50:54.335740 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.336142 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.336152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.336154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.336279 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:50:54.336553 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:50:54.336556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:50:54.336558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:50:54.336733 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:50:54.336869 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.337092 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.337206 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:50:54.339959 16315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:50:54.343760 16315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:50:54.352746 16437 tablet_service.cc:1467] Processing CreateTablet for tablet a6190a0317de451e99b93a5ba9630f05 (DEFAULT_TABLE table=test-table [id=482ef4f568044d2f81256b81414b6c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:54.353412 16437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6190a0317de451e99b93a5ba9630f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:54.356487 16395 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap starting.
I20260929 14:50:54.357306 16395 tablet_bootstrap.cc:654] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:54.357393 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.351391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33717 (local address 127.0.0.1:54150)
0929 14:50:54.351533 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.351552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.351651 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.352940 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.352944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.352962 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:54.353115 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.353122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.354399 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.354404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.355535 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.355547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.356883 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.356886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.357005 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.357065 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.357123 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.357220 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":75}
I20260929 14:50:54.358150 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.351931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.0.98.3:33717)
0929 14:50:54.352415 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.352436 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.352463 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:50:54.352549 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.352554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.352776 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.353107 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:54.353340 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.354274 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.355654 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.356749 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.357623 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.357795 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.357857 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.357969 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":183,"threads_started":1}
I20260929 14:50:54.358281 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.351667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.0.1:35722)
0929 14:50:54.352450 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.352467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.352577 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.353510 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.353513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.353531 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:54.353645 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.353651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.355218 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.355222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.356478 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.356490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.357860 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.357865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.357962 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.357983 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.358067 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.358136 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":36,"thread_start_us":159,"threads_started":1}
I20260929 14:50:54.358958 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.351941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.0.98.2:36165)
0929 14:50:54.352891 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.352906 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.352988 (+ 82us) server_negotiation.cc:407] Connection header received
0929 14:50:54.353072 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.353077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.353334 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.353808 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:54.353987 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.355074 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.356653 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.357528 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.357631 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.357795 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.358141 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.358742 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":193,"threads_started":1}
I20260929 14:50:54.359191 16395 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: No bootstrap required, opened a new log
I20260929 14:50:54.359287 16395 ts_tablet_manager.cc:1397] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:50:54.359689 16395 raft_consensus.cc:348] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.359823 16395 raft_consensus.cc:374] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:54.359869 16395 raft_consensus.cc:729] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633f9e7d0b9d4f738f3eb09163b94a82, State: Initialized, Role: FOLLOWER
I20260929 14:50:54.359994 16395 consensus_queue.cc:260] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.360299 16395 ts_tablet_manager.cc:1428] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:50:54.360399 16576 tablet_service.cc:1467] Processing CreateTablet for tablet a6190a0317de451e99b93a5ba9630f05 (DEFAULT_TABLE table=test-table [id=482ef4f568044d2f81256b81414b6c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:54.361711 16576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6190a0317de451e99b93a5ba9630f05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:50:54.361948 16708 tablet_service.cc:1467] Processing CreateTablet for tablet a6190a0317de451e99b93a5ba9630f05 (DEFAULT_TABLE table=test-table [id=482ef4f568044d2f81256b81414b6c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:50:54.362910 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6190a0317de451e99b93a5ba9630f05. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:50:54.366030 16504 tablet.cc:2369] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:50:54.367663 16534 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Bootstrap starting.
I20260929 14:50:54.369977 16534 tablet_bootstrap.cc:654] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:54.370862 16534 log.cc:824] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:54.373347 16534 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: No bootstrap required, opened a new log
I20260929 14:50:54.373584 16534 ts_tablet_manager.cc:1397] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:50:54.375773 16788 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Bootstrap starting.
I20260929 14:50:54.377166 16534 raft_consensus.cc:348] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.377445 16534 raft_consensus.cc:374] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:54.377508 16534 raft_consensus.cc:729] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef70f7de9db431b804604a43dfef81f, State: Initialized, Role: FOLLOWER
I20260929 14:50:54.377976 16534 consensus_queue.cc:260] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.378293 16788 tablet_bootstrap.cc:654] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Neither blocks nor log segments found. Creating new log.
I20260929 14:50:54.379132 16642 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
I20260929 14:50:54.379237 16534 ts_tablet_manager.cc:1428] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:50:54.379669 16788 log.cc:824] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:50:54.382176 16788 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: No bootstrap required, opened a new log
I20260929 14:50:54.382416 16788 ts_tablet_manager.cc:1397] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:50:54.385720 16788 raft_consensus.cc:348] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.385987 16788 raft_consensus.cc:374] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:50:54.386040 16788 raft_consensus.cc:729] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5426692ebc73469489d7f231b2846cf1, State: Initialized, Role: FOLLOWER
I20260929 14:50:54.386530 16788 consensus_queue.cc:260] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.387300 16774 heartbeater.cc:503] Master 127.0.98.62:36921 was elected leader, sending a full tablet report...
I20260929 14:50:54.387543 16788 ts_tablet_manager.cc:1428] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:50:54.533708 16789 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:50:54.533941 16789 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.535061 16789 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543)
I20260929 14:50:54.539914 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.535493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33717 (local address 127.0.98.2:50789)
0929 14:50:54.535731 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.535750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.535893 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.536401 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.536406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.536430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:54.536637 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.536644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.537594 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.537597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.538573 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.538584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.539430 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.539434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.539549 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.539578 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.539671 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.539751 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":24,"spinlock_wait_cycles":14208}
I20260929 14:50:54.540035 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.535650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50789 (local address 127.0.98.3:33717)
0929 14:50:54.535843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.535849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.535866 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:50:54.536085 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.536092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.536202 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.536303 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:54.536773 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.537470 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.538720 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.539292 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.539379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.539446 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.539740 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.539865 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 14:50:54.540386 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.535878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39543 (local address 127.0.98.2:47069)
0929 14:50:54.536385 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.536401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:50:54.536555 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:50:54.536918 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:50:54.536921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:50:54.536939 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:50:54.537086 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.537092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.537980 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.537983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.539001 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:50:54.539011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.539870 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.539873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:50:54.539990 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.540016 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:50:54.540114 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:50:54.540222 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":3,"spinlock_wait_cycles":29056,"thread_start_us":185,"threads_started":1}
I20260929 14:50:54.540511 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.536079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47069 (local address 127.0.98.1:39543)
0929 14:50:54.536392 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.536397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.536493 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:50:54.536667 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.536672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.536755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.536877 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:50:54.537202 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.537852 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.539130 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.539725 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.539840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.540005 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.540283 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.540363 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":154,"threads_started":1}
I20260929 14:50:54.540891 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6190a0317de451e99b93a5ba9630f05" candidate_uuid: "9ef70f7de9db431b804604a43dfef81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426692ebc73469489d7f231b2846cf1" is_pre_election: true
I20260929 14:50:54.541141 16457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6190a0317de451e99b93a5ba9630f05" candidate_uuid: "9ef70f7de9db431b804604a43dfef81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" is_pre_election: true
I20260929 14:50:54.541350 16728 raft_consensus.cc:2454] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef70f7de9db431b804604a43dfef81f in term 0.
I20260929 14:50:54.541435 16457 raft_consensus.cc:2454] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef70f7de9db431b804604a43dfef81f in term 0.
I20260929 14:50:54.541949 16530 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82, 9ef70f7de9db431b804604a43dfef81f; no voters:
I20260929 14:50:54.542237 16789 raft_consensus.cc:2781] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:50:54.542336 16789 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:50:54.542384 16789 raft_consensus.cc:3037] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:50:54.542899 16775 tablet.cc:2369] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:50:54.543767 16789 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.544166 16789 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [CANDIDATE]: Term 1 election: Requested vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543)
I20260929 14:50:54.544710 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6190a0317de451e99b93a5ba9630f05" candidate_uuid: "9ef70f7de9db431b804604a43dfef81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5426692ebc73469489d7f231b2846cf1"
I20260929 14:50:54.544719 16457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6190a0317de451e99b93a5ba9630f05" candidate_uuid: "9ef70f7de9db431b804604a43dfef81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
I20260929 14:50:54.544881 16457 raft_consensus.cc:3037] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:54.544891 16728 raft_consensus.cc:3037] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:50:54.546159 16457 raft_consensus.cc:2454] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef70f7de9db431b804604a43dfef81f in term 1.
I20260929 14:50:54.546159 16728 raft_consensus.cc:2454] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef70f7de9db431b804604a43dfef81f in term 1.
I20260929 14:50:54.546500 16530 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f; no voters:
I20260929 14:50:54.546686 16789 raft_consensus.cc:2781] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:50:54.546931 16789 raft_consensus.cc:686] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 1 LEADER]: Becoming Leader. State: Replica: 9ef70f7de9db431b804604a43dfef81f, State: Running, Role: LEADER
I20260929 14:50:54.547278 16789 consensus_queue.cc:237] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:50:54.551182 16315 catalog_manager.cc:5495] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef70f7de9db431b804604a43dfef81f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9ef70f7de9db431b804604a43dfef81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } health_report { overall_health: UNKNOWN } } }
W20260929 14:50:54.583814 16643 tablet.cc:2369] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:50:54.602188 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.598385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.0.98.2:36165)
0929 14:50:54.598547 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:50:54.598551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:50:54.598568 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:50:54.598706 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:50:54.598708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:50:54.598770 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:50:54.598872 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:50:54.599200 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.599756 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.600734 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:50:54.600955 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:50:54.601048 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:50:54.601351 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:50:54.601754 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:50:54.601849 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:50:54.601949 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:50:54.601999 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 14:50:54.609141 16728 raft_consensus.cc:1261] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Refusing update from remote peer 9ef70f7de9db431b804604a43dfef81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:54.609140 16457 raft_consensus.cc:1261] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Refusing update from remote peer 9ef70f7de9db431b804604a43dfef81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:50:54.609966 16789 consensus_queue.cc:1035] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:54.610175 16794 consensus_queue.cc:1035] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:50:54.618636 16799 mvcc.cc:205] Tried to move back new op lower bound from 7334680390069850112 to 7334680389828673536. Current Snapshot: MvccSnapshot[applied={T|T < 7334680390069850112}]
I20260929 14:50:54.620858 16798 mvcc.cc:205] Tried to move back new op lower bound from 7334680390069850112 to 7334680389828673536. Current Snapshot: MvccSnapshot[applied={T|T < 7334680390069850112 or (T in {7334680390069850112})}]
I20260929 14:50:54.628615 16315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:54.628799 16315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:50:54.631587 16315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Sending DeleteTablet for 3 replicas of tablet a6190a0317de451e99b93a5ba9630f05
I20260929 14:50:54.632382 16708 tablet_service.cc:1514] Processing DeleteTablet for tablet a6190a0317de451e99b93a5ba9630f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54150
I20260929 14:50:54.632546 16576 tablet_service.cc:1514] Processing DeleteTablet for tablet a6190a0317de451e99b93a5ba9630f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35722
I20260929 14:50:54.632697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16353
I20260929 14:50:54.632654 16437 tablet_service.cc:1514] Processing DeleteTablet for tablet a6190a0317de451e99b93a5ba9630f05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:50:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54434
I20260929 14:50:54.634591 16807 tablet_replica.cc:330] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: stopping tablet replica
I20260929 14:50:54.634872 16808 tablet_replica.cc:330] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: stopping tablet replica
I20260929 14:50:54.635180 16808 raft_consensus.cc:2229] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:50:54.635555 16808 raft_consensus.cc:2258] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:54.636041 16807 raft_consensus.cc:2229] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:50:54.636351 16807 raft_consensus.cc:2258] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:50:54.637625 16807 ts_tablet_manager.cc:1905] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:50:54.639961 16807 ts_tablet_manager.cc:1918] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:54.640048 16807 log.cc:1196] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/a6190a0317de451e99b93a5ba9630f05
I20260929 14:50:54.641240 16808 ts_tablet_manager.cc:1905] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:50:54.642954 16301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39543 (error 108)
I20260929 14:50:54.643920 16808 ts_tablet_manager.cc:1918] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:50:54.643986 16808 log.cc:1196] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/a6190a0317de451e99b93a5ba9630f05
I20260929 14:50:54.644434 16807 ts_tablet_manager.cc:1939] T a6190a0317de451e99b93a5ba9630f05 P 5426692ebc73469489d7f231b2846cf1: Deleting consensus metadata
I20260929 14:50:54.644408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16507
I20260929 14:50:54.645689 16301 catalog_manager.cc:4841] TS 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): tablet a6190a0317de451e99b93a5ba9630f05 (table test-table [id=482ef4f568044d2f81256b81414b6c2a]) successfully deleted
I20260929 14:50:54.646871 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.646503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39543 (local address 127.0.0.1:54436)
0929 14:50:54.646670 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.646764 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:50:54.647259 16301 catalog_manager.cc:4584] TS 633f9e7d0b9d4f738f3eb09163b94a82 (127.0.98.1:39543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6190a0317de451e99b93a5ba9630f05: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39543: connect: Connection refused (error 111)
I20260929 14:50:54.652671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16645
I20260929 14:50:54.652846 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:50:54.652520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.0.1:35738)
0929 14:50:54.652655 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 14:50:54.652719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 14:50:54.660301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16282
I20260929 14:51:14.668051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--webserver_interface=127.0.98.62
--webserver_port=43331
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:14.781293 16810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:14.781569 16810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:14.781608 16810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:14.785992 16810 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:51:14.786064 16810 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:51:14.786087 16810 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:51:14.786100 16810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:14.786119 16810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:14.786137 16810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:14.786154 16810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:14.786211 16810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:14.791533 16810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:14.792742 16810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:14.793831 16810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:14.804136 16815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:14.804340 16810 server_base.cc:1034] running on GCE node
W20260929 14:51:14.804354 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:14.804366 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:14.805361 16810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:14.806962 16810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:14.808269 16810 hybrid_clock.cc:648] HybridClock initialized: now 1790693474808234 us; error 62 us; skew 500 ppm
Sep 29 14:51:14 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:14.817364 16810 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:51:14.819063 16810 webserver.cc:460] Webserver started at http://127.0.98.62:43331/ using document root <none> and password file <none>
I20260929 14:51:14.819886 16810 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:14.820003 16810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:14.825608 16810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:51:14.828562 16825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:14.830658 16810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:14.830888 16810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "9e8bf2f9df4e48aba347d9f52c182ef1"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "8aa54862591f3923a512a632d5a33536"
server_key_iv: "346423046e42bcea2946d9bc73b0a542"
server_key_version: "encryptionkey@0"
I20260929 14:51:14.831655 16810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:14.852568 16810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:14.857345 16810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:14.857620 16810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:14.870499 16810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36921
I20260929 14:51:14.870816 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36921 every 8 connection(s)
I20260929 14:51:14.872728 16810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:14.876916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16810
I20260929 14:51:14.877583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39543
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=42793
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:51:14.878423 16878 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:51:14.880021 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap starting.
I20260929 14:51:14.897495 16878 log.cc:824] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:14.908947 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:51:14.909334 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Bootstrap complete.
I20260929 14:51:14.913846 16878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:51:14.914331 16878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Initialized, Role: FOLLOWER
I20260929 14:51:14.914901 16878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:51:14.915081 16878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:14.915185 16878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:14.915284 16878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:51:14.917412 16878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:51:14.917768 16878 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8bf2f9df4e48aba347d9f52c182ef1; no voters:
I20260929 14:51:14.918118 16878 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:51:14.918396 16882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:51:14.918864 16882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [term 3 LEADER]: Becoming Leader. State: Replica: 9e8bf2f9df4e48aba347d9f52c182ef1, State: Running, Role: LEADER
I20260929 14:51:14.919338 16882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } }
I20260929 14:51:14.919533 16878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:14.921079 16884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8bf2f9df4e48aba347d9f52c182ef1. Latest consensus state: current_term: 3 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:51:14.921226 16884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:14.921579 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8bf2f9df4e48aba347d9f52c182ef1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36921 } } }
I20260929 14:51:14.921622 16894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:14.921663 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:14.926957 16894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:14.927608 16894 catalog_manager.cc:1259] Loaded cluster ID: cbf09167d204421eb3aa1456f3115d43
I20260929 14:51:14.927673 16894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:14.930253 16894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:14.930989 16894 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Loaded TSK: 0
I20260929 14:51:14.931077 16894 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e8bf2f9df4e48aba347d9f52c182ef1: Loaded TSK: 1
F20260929 14:51:14.932296 16894 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4b365e69ad google::LogMessage::Fail() at ??:0
@ 0x7f4b365e988a google::LogMessage::SendToLog() at ??:0
@ 0x7f4b365e64de google::LogMessage::Flush() at ??:0
@ 0x7f4b365e7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4b392aa18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4b392a2f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4b40f4b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4b40f2599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4b40f26178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4b40f67df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4b40f8682b std::function<>::operator()() at ??:0
@ 0x7f4b40f25b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4b40f26f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4b40f23579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4b40f67640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4b388383d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4b38838c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4b3883a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4b38829512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4b39e6d6db start_thread at ??:0
@ 0x7f4b350e461f clone at ??:0
W20260929 14:51:15.061080 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:15.061482 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:15.061592 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:15.067073 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:15.067222 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:15.067335 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:15.075335 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42793
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:15.077024 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:15.078543 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:15.091962 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.091988 16928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.091988 16926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:15.092423 16880 server_base.cc:1034] running on GCE node
I20260929 14:51:15.093199 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:15.094157 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:15.095359 16880 hybrid_clock.cc:648] HybridClock initialized: now 1790693475095326 us; error 50 us; skew 500 ppm
Sep 29 14:51:15 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:15.099298 16880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:51:15.100685 16880 webserver.cc:460] Webserver started at http://127.0.98.1:42793/ using document root <none> and password file <none>
I20260929 14:51:15.101413 16880 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:15.101722 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:15.106827 16880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:51:15.109905 16935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.111754 16880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:15.111995 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "633f9e7d0b9d4f738f3eb09163b94a82"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "00f617bde105bcd5ab3a6ec70e47416a"
server_key_iv: "04ef1a8ab271a1a64c148258191d9970"
server_key_version: "encryptionkey@0"
I20260929 14:51:15.112793 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:15.133195 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:15.137293 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:15.137547 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:15.138314 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:15.139755 16942 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:51:15.145706 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:51:15.145872 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 14:51:15.145975 16880 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:51:15.149344 16880 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:51:15.149451 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:15.149821 16942 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap starting.
I20260929 14:51:15.162819 16880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39543
I20260929 14:51:15.162953 17049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39543 every 8 connection(s)
I20260929 14:51:15.164085 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:15.166577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 16880
I20260929 14:51:15.167184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36165
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=40895
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:51:15.169157 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.166779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:58237)
0929 14:51:15.167819 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.167999 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":178,"threads_started":1}
W20260929 14:51:15.169771 17050 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36921 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36921: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
I20260929 14:51:15.169880 16942 log.cc:824] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:15.170828 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.170437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:37129)
0929 14:51:15.170634 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.170715 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 14:51:15.171743 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.171494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:48713)
0929 14:51:15.171628 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.171684 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 14:51:15.171991 17050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:51:15.177379 16942 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:51:15.177755 16942 tablet_bootstrap.cc:492] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Bootstrap complete.
I20260929 14:51:15.178956 16942 ts_tablet_manager.cc:1397] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.011s
I20260929 14:51:15.181703 16942 raft_consensus.cc:348] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:15.182160 16942 raft_consensus.cc:729] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633f9e7d0b9d4f738f3eb09163b94a82, State: Initialized, Role: FOLLOWER
I20260929 14:51:15.182614 16942 consensus_queue.cc:260] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:15.183655 16942 ts_tablet_manager.cc:1428] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260929 14:51:15.184157 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.183778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:36687)
0929 14:51:15.183943 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.184039 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 14:51:15.184945 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.184711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:47855)
0929 14:51:15.184832 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.184884 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 14:51:15.305481 17053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:15.305860 17053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:15.305948 17053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:15.310830 17053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:15.310950 17053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:15.311093 17053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:15.318058 17053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40895
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:15.319693 17053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:15.321043 17053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:15.332746 17061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.332739 17060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.332746 17063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:15.333119 17053 server_base.cc:1034] running on GCE node
I20260929 14:51:15.333645 17053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:15.334376 17053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:15.335556 17053 hybrid_clock.cc:648] HybridClock initialized: now 1790693475335542 us; error 35 us; skew 500 ppm
Sep 29 14:51:15 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:15.338732 17053 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:51:15.339810 17053 webserver.cc:460] Webserver started at http://127.0.98.2:40895/ using document root <none> and password file <none>
I20260929 14:51:15.340551 17053 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:15.340624 17053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:15.345337 17053 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:51:15.348348 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.349967 17053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:15.350128 17053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "9ef70f7de9db431b804604a43dfef81f"
format_stamp: "Formatted at 2026-09-29 14:50:32 on dist-test-slave-5hd1"
server_key: "ff3d3bab11dc7212d70df4c7bcb5b055"
server_key_iv: "59a45449e671bc3dc810864a606cf25b"
server_key_version: "encryptionkey@0"
I20260929 14:51:15.350813 17053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:15.368400 17053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:15.372367 17053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:15.372561 17053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:15.373246 17053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:15.374519 17077 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 14:51:15.378746 17077 ts_tablet_manager.cc:1852] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 14:51:15.379161 17077 ts_tablet_manager.cc:1905] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:15.382913 17077 ts_tablet_manager.cc:1918] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:15.383013 17077 log.cc:1196] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:15.383292 17077 ts_tablet_manager.cc:1939] T a6190a0317de451e99b93a5ba9630f05 P 9ef70f7de9db431b804604a43dfef81f: Deleting consensus metadata
I20260929 14:51:15.383881 17053 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 14:51:15.384037 17053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20260929 14:51:15.384151 17053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:15.384244 17053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.395714 17053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36165
I20260929 14:51:15.395742 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36165 every 8 connection(s)
I20260929 14:51:15.396883 17053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:15.397877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17053
I20260929 14:51:15.398401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33717
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36921
--webserver_port=38363
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:51:15.401651 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.399679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:40963)
0929 14:51:15.400411 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.400617 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":165,"threads_started":1}
W20260929 14:51:15.402231 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36921 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36921: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
I20260929 14:51:15.403062 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.402789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:37699)
0929 14:51:15.402952 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.403010 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 14:51:15.403940 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.403703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:49587)
0929 14:51:15.403843 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.403893 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 14:51:15.404250 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:51:15.532109 17188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:15.532532 17188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:15.532620 17188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:15.537040 17188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:15.537137 17188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:15.537238 17188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:15.543044 17188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38363
--tserver_master_addrs=127.0.98.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:15.544492 17188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:15.545804 17188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:15.558420 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.558447 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:15.558776 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:15.558923 17188 server_base.cc:1034] running on GCE node
I20260929 14:51:15.559363 17188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:15.560082 17188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:15.561283 17188 hybrid_clock.cc:648] HybridClock initialized: now 1790693475561253 us; error 50 us; skew 500 ppm
Sep 29 14:51:15 dist-test-slave-5hd1 krb5kdc[15734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:15.564386 17188 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:51:15.565479 17188 webserver.cc:460] Webserver started at http://127.0.98.3:38363/ using document root <none> and password file <none>
I20260929 14:51:15.566188 17188 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:15.566258 17188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:15.570922 17188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 14:51:15.573627 17203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.575137 17188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:15.575299 17188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "5426692ebc73469489d7f231b2846cf1"
format_stamp: "Formatted at 2026-09-29 14:50:33 on dist-test-slave-5hd1"
server_key: "454113ac6edc0d8b015495da25d72a2e"
server_key_iv: "17ee9290354a345246cc86050e189a9b"
server_key_version: "encryptionkey@0"
I20260929 14:51:15.575999 17188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:15.597219 17188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:15.601198 17188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:15.601470 17188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:15.602417 17188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:15.603492 17188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:15.603567 17188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.603646 17188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:15.603693 17188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:15.616389 17188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33717
I20260929 14:51:15.616412 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33717 every 8 connection(s)
I20260929 14:51:15.617717 17188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:15.618636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17188
I20260929 14:51:15.622776 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.620565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:45375)
0929 14:51:15.621201 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.621338 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":191,"threads_started":1}
W20260929 14:51:15.623308 17317 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36921 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36921: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
I20260929 14:51:15.623921 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.623679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:53819)
0929 14:51:15.623805 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.623861 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:51:15.624676 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:15.624484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:48511)
0929 14:51:15.624583 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:15.624626 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:51:15.624893 17317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:51:16.186350 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.185764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:57797)
0929 14:51:16.186131 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:16.186219 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260929 14:51:16.405390 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.404878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:38703)
0929 14:51:16.405192 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:16.405264 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260929 14:51:16.439564 17323 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:16.439860 17323 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:16.441148 17323 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
Sep 29 14:51:16 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 14:51:16 dist-test-slave-5hd1 krb5kdc[15734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 14:51:16.459000 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.441569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33717 (local address 127.0.98.1:59355)
0929 14:51:16.441817 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:16.441867 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:16.442555 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:16.444621 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:16.444635 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:16.445277 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:16.445821 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:16.445839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.447222 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.447228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:16.448402 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:16.448413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.449361 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.449365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:16.449631 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:16.450440 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:16.450453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:16.452366 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:16.455291 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:16.455298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:16.455316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:16.455946 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:16.456547 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:16.456553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:16.456789 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:16.457710 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:16.457717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:16.458115 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 14:51:16.458349 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 14:51:16.458826 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":352,"spinlock_wait_cycles":27776}
I20260929 14:51:16.459134 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.441820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59355 (local address 127.0.98.3:33717)
0929 14:51:16.442541 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:16.442563 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:16.442636 (+ 73us) server_negotiation.cc:407] Connection header received
0929 14:51:16.443765 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:16.443772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:16.444140 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:16.444502 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:16.446022 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.447051 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.448552 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.449213 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.449427 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:16.452595 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:16.452600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:16.452610 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:16.452652 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:16.455137 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456215 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:16.456221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:16.456226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:16.456426 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456904 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:16.456907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:16.456909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:16.457568 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:16.457725 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:16.458523 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 14:51:16.458899 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":170,"threads_started":1}
I20260929 14:51:16.459129 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.442428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35309 (local address 127.0.98.2:36165)
0929 14:51:16.442920 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:16.442945 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:16.443058 (+ 113us) server_negotiation.cc:407] Connection header received
0929 14:51:16.443955 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:16.443962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:16.444392 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:16.444743 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:16.446017 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.447055 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.448595 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.449246 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.449437 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:16.452512 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:16.452521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:16.452541 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:16.452585 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:16.455137 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456211 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:16.456219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:16.456226 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:16.456426 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456904 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:16.456907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:16.456909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:16.457568 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:16.457725 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:16.458523 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 14:51:16.458899 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":110,"threads_started":1}
I20260929 14:51:16.460322 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.441895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36165 (local address 127.0.98.1:35309)
0929 14:51:16.442904 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:16.442924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:16.443088 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:16.444740 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:16.444744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:16.445258 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:16.445821 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:16.445838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.447220 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.447223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:16.448411 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:16.448422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:16.449389 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:16.449393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:16.449640 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:16.450265 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:16.450281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:16.452128 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:16.455291 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:16.455298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:16.455316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:16.455946 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:16.456547 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:16.456550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:16.456553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:16.456789 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:16.457710 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:16.457717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:16.458115 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 14:51:16.458349 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 14:51:16.458774 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":339,"spinlock_wait_cycles":51072,"thread_start_us":109,"threads_started":1}
W20260929 14:51:16.462333 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
W20260929 14:51:16.462896 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:16.462966 16938 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82; no voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f
I20260929 14:51:16.463191 17323 raft_consensus.cc:2726] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:51:16.626189 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:16.625639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:46829)
0929 14:51:16.625959 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:16.626047 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260929 14:51:17.188004 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:17.187449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:54733)
0929 14:51:17.187817 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:17.187901 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":105,"threads_started":1}
I20260929 14:51:17.406829 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:17.406324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:60003)
0929 14:51:17.406615 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:17.406685 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260929 14:51:17.627805 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:17.627208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:49689)
0929 14:51:17.627563 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:17.627653 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260929 14:51:17.989519 17331 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:17.989696 17331 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:17.990134 17331 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
W20260929 14:51:17.991201 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
W20260929 14:51:17.991391 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:17.991460 16938 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82; no voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f
I20260929 14:51:17.991665 17331 raft_consensus.cc:2726] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:51:18.189750 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:18.189131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:56153)
0929 14:51:18.189558 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:18.189636 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":105,"threads_started":1}
I20260929 14:51:18.408370 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:18.407848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:56543)
0929 14:51:18.408127 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:18.408237 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260929 14:51:18.629400 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:18.628843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:37345)
0929 14:51:18.629186 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:18.629275 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260929 14:51:19.191450 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:19.190867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:34451)
0929 14:51:19.191253 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:19.191336 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260929 14:51:19.410055 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:19.409505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:59543)
0929 14:51:19.409855 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:19.409928 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260929 14:51:19.630941 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:19.630372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:57089)
0929 14:51:19.630723 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:19.630813 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":83,"threads_started":1}
I20260929 14:51:20.142537 17338 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:20.142688 17338 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:20.143050 17338 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
W20260929 14:51:20.144394 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
W20260929 14:51:20.144598 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:20.144665 16938 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82; no voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f
I20260929 14:51:20.144862 17338 raft_consensus.cc:2726] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:51:20.193045 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:20.192580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:58975)
0929 14:51:20.192881 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:20.192948 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
I20260929 14:51:20.411639 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:20.411133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:50905)
0929 14:51:20.411456 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:20.411522 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260929 14:51:20.632495 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:20.631939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:49199)
0929 14:51:20.632296 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:20.632373 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260929 14:51:21.194583 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:21.194038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:56773)
0929 14:51:21.194374 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:21.194451 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260929 14:51:21.413256 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:21.412740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:53575)
0929 14:51:21.413069 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:21.413153 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260929 14:51:21.633961 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:21.633394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:56515)
0929 14:51:21.633728 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:21.633802 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260929 14:51:22.196018 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:22.195518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:56185)
0929 14:51:22.195841 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:22.195912 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260929 14:51:22.414706 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:22.414037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:36635)
0929 14:51:22.414446 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:22.414523 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":76,"threads_started":1}
I20260929 14:51:22.635540 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:22.634916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:34199)
0929 14:51:22.635342 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:22.635426 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":126,"threads_started":1}
I20260929 14:51:23.197583 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:23.197015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:46947)
0929 14:51:23.197387 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:23.197468 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":96,"threads_started":1}
I20260929 14:51:23.242959 17349 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:23.243139 17349 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:23.243712 17349 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
W20260929 14:51:23.244648 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
W20260929 14:51:23.244804 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:23.244867 16938 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82; no voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f
I20260929 14:51:23.245005 17349 raft_consensus.cc:2726] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:51:23.416251 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:23.415611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:34233)
0929 14:51:23.415955 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:23.416040 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":82,"threads_started":1}
I20260929 14:51:23.637092 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:23.636527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:60341)
0929 14:51:23.636875 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:23.636959 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260929 14:51:24.199174 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:24.198575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:59053)
0929 14:51:24.198988 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:24.199053 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":200,"threads_started":1}
I20260929 14:51:24.417724 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:24.417212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:34031)
0929 14:51:24.417522 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:24.417595 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260929 14:51:24.638733 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:24.638142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:54165)
0929 14:51:24.638503 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:24.638591 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
I20260929 14:51:25.200905 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:25.200374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:37565)
0929 14:51:25.200719 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:25.200800 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260929 14:51:25.419327 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:25.418659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:56127)
0929 14:51:25.419052 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:25.419139 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260929 14:51:25.640367 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:25.639755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:47497)
0929 14:51:25.640089 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:25.640164 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260929 14:51:26.202545 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:26.201980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:42057)
0929 14:51:26.202365 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:26.202438 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260929 14:51:26.420845 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:26.420305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:47571)
0929 14:51:26.420628 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:26.420713 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260929 14:51:26.465260 17360 raft_consensus.cc:482] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:26.465417 17360 raft_consensus.cc:504] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5426692ebc73469489d7f231b2846cf1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33717 } } peers { permanent_uuid: "9ef70f7de9db431b804604a43dfef81f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36165 } } peers { permanent_uuid: "633f9e7d0b9d4f738f3eb09163b94a82" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39543 } }
I20260929 14:51:26.465911 17360 leader_election.cc:288] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717), 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165)
W20260929 14:51:26.466923 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ef70f7de9db431b804604a43dfef81f (127.0.98.2:36165): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
W20260929 14:51:26.467073 16938 leader_election.cc:341] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5426692ebc73469489d7f231b2846cf1 (127.0.98.3:33717): Not found: Tablet not found: a6190a0317de451e99b93a5ba9630f05
I20260929 14:51:26.467142 16938 leader_election.cc:302] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 633f9e7d0b9d4f738f3eb09163b94a82; no voters: 5426692ebc73469489d7f231b2846cf1, 9ef70f7de9db431b804604a43dfef81f
I20260929 14:51:26.467357 17360 raft_consensus.cc:2726] T a6190a0317de451e99b93a5ba9630f05 P 633f9e7d0b9d4f738f3eb09163b94a82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:51:26.641914 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:26.641315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:40477)
0929 14:51:26.641699 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:26.641797 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":111,"threads_started":1}
I20260929 14:51:27.203997 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:27.203414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:35749)
0929 14:51:27.203769 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:27.203855 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
I20260929 14:51:27.422307 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:27.421731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:37021)
0929 14:51:27.422081 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:27.422167 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
I20260929 14:51:27.643329 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:27.642819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:35553)
0929 14:51:27.643134 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:27.643219 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260929 14:51:28.205574 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:28.205006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:59269)
0929 14:51:28.205377 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:28.205461 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260929 14:51:28.423671 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:28.423165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:51199)
0929 14:51:28.423492 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:28.423557 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":120,"threads_started":1}
I20260929 14:51:28.644812 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:28.644245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:55353)
0929 14:51:28.644580 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:28.644667 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260929 14:51:29.207039 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:29.206567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:39459)
0929 14:51:29.206880 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:29.206937 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260929 14:51:29.425173 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:29.424654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:50625)
0929 14:51:29.424997 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:29.425069 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260929 14:51:29.646287 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:29.645782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.3:60527)
0929 14:51:29.646118 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:29.646181 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260929 14:51:30.208491 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:30.208075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.1:55677)
0929 14:51:30.208352 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:30.208396 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260929 14:51:30.426725 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:30.426209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36921 (local address 127.0.98.2:40861)
0929 14:51:30.426545 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:30.426620 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260929 14:51:30.628302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 16880
I20260929 14:51:30.635324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17053
I20260929 14:51:30.641338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17188
2026-09-29T14:51:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59661 ms)
[----------] 30 tests from SecurityITest (252100 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:30 dist-test-slave-5hd1 krb5kdc[17375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:30 dist-test-slave-5hd1 krb5kdc[17375](info): set up 2 sockets
Sep 29 14:51:30 dist-test-slave-5hd1 krb5kdc[17375](info): commencing operation
krb5kdc: starting...
W20260929 14:51:31.700553 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:31 dist-test-slave-5hd1 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:51:31.840406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:31.973730 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:31.974073 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:31.974120 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:31.978457 17388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:31.978529 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:31.978549 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:31.978569 17388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:31.978583 17388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:31.984338 17388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:31.985749 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:31.986992 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:31.997519 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:31.997594 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:31.997568 17388 server_base.cc:1034] running on GCE node
W20260929 14:51:31.997503 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:31.998607 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:31.999837 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:32.001037 17388 hybrid_clock.cc:648] HybridClock initialized: now 1790693492000985 us; error 63 us; skew 500 ppm
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:32.004779 17388 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:51:32.005859 17388 webserver.cc:460] Webserver started at http://127.0.98.62:44409/ using document root <none> and password file <none>
I20260929 14:51:32.006513 17388 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:32.006610 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:32.006857 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:32.009380 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "30f3cabc38e34e01b398b99eb2b7cccc"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "c99870f159984092d865ba5a5bd50388"
server_key_iv: "b53c16fdc6ce9b1ea817c7df9975a0c9"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.010033 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "30f3cabc38e34e01b398b99eb2b7cccc"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "c99870f159984092d865ba5a5bd50388"
server_key_iv: "b53c16fdc6ce9b1ea817c7df9975a0c9"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.014482 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 14:51:32.017586 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.019196 17388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:32.019369 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "30f3cabc38e34e01b398b99eb2b7cccc"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "c99870f159984092d865ba5a5bd50388"
server_key_iv: "b53c16fdc6ce9b1ea817c7df9975a0c9"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.019505 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:32.035251 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:32.038800 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:32.039021 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:32.047178 17388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43655
I20260929 14:51:32.047237 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43655 every 8 connection(s)
I20260929 14:51:32.048314 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:32.048693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17388
I20260929 14:51:32.048882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:32.049254 392 external_mini_cluster.cc:1428] Setting key e3b25adb73b26ab8f24f907071ff29a2
I20260929 14:51:32.051420 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:32.058416 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: Bootstrap starting.
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:32.061020 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:32.061950 17456 log.cc:824] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:32.063901 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: No bootstrap required, opened a new log
I20260929 14:51:32.066278 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.051069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.0.98.62:43655)
0929 14:51:32.051565 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.051589 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.051642 (+ 53us) server_negotiation.cc:407] Connection header received
0929 14:51:32.052500 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.052523 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.052933 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.053279 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:32.054291 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.055454 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.056717 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.057289 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.057433 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.060847 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:32.060853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:32.060873 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:32.060910 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:32.063009 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.063502 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.063512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.063522 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.063647 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.063940 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.063945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.063947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.064508 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:32.064730 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.065025 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.065385 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":176,"threads_started":1}
I20260929 14:51:32.067731 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260929 14:51:32.068001 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:32.068075 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3cabc38e34e01b398b99eb2b7cccc, State: Initialized, Role: FOLLOWER
I20260929 14:51:32.068560 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260929 14:51:32.068747 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:32.068856 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:32.068969 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:32.069860 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260929 14:51:32.070195 17456 leader_election.cc:302] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f3cabc38e34e01b398b99eb2b7cccc; no voters:
I20260929 14:51:32.070498 17456 leader_election.cc:288] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:32.070712 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:32.071018 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [term 1 LEADER]: Becoming Leader. State: Replica: 30f3cabc38e34e01b398b99eb2b7cccc, State: Running, Role: LEADER
I20260929 14:51:32.071399 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } }
I20260929 14:51:32.071723 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:32.073411 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f3cabc38e34e01b398b99eb2b7cccc. Latest consensus state: current_term: 1 leader_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } }
I20260929 14:51:32.073558 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:32.073910 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f3cabc38e34e01b398b99eb2b7cccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43655 } } }
I20260929 14:51:32.073993 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:32.073992 17473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:32.077951 17473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:32.083290 17473 catalog_manager.cc:1347] Generated new cluster ID: 2c9288b93bab45b58fd7aaefb82d2454
I20260929 14:51:32.083364 17473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:32.090504 17473 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:32.092051 17473 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:32.103600 17473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: Generated new TSK 0
I20260929 14:51:32.104519 17473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:51:32.141633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:32.261520 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:32.261803 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:32.261847 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:32.265890 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:32.265959 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:32.266064 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:32.271582 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:32.272864 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:32.273936 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:32.284598 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:32.284555 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:32.284576 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:32.284929 17482 server_base.cc:1034] running on GCE node
I20260929 14:51:32.285518 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:32.286258 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:32.287442 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790693492287417 us; error 40 us; skew 500 ppm
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:32.290941 17482 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:51:32.292040 17482 webserver.cc:460] Webserver started at http://127.0.98.1:43109/ using document root <none> and password file <none>
I20260929 14:51:32.292779 17482 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:32.292855 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:32.293104 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:32.295539 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "654addf9b4c4456c9f7284510db8fb7e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "9fd856711e934f2c9eec0d2cf53464a7"
server_key_iv: "529c2cdfb47652b4d93c7c2248a17d59"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.296231 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "654addf9b4c4456c9f7284510db8fb7e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "9fd856711e934f2c9eec0d2cf53464a7"
server_key_iv: "529c2cdfb47652b4d93c7c2248a17d59"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.300799 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:32.303855 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.305575 17482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:32.305754 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "654addf9b4c4456c9f7284510db8fb7e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "9fd856711e934f2c9eec0d2cf53464a7"
server_key_iv: "529c2cdfb47652b4d93c7c2248a17d59"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.305896 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:32.324029 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:32.328953 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:32.329210 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:32.330175 17482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:32.331624 17482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:32.331700 17482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.331771 17482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:32.331812 17482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.347570 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37071
I20260929 14:51:32.347606 17610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37071 every 8 connection(s)
I20260929 14:51:32.349349 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:32.359079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17482
I20260929 14:51:32.359238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:32.359596 392 external_mini_cluster.cc:1428] Setting key b5f27c5b34b96506b4c62706df1e4e8d
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:32.366572 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.351820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34835 (local address 127.0.98.62:43655)
0929 14:51:32.351965 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.351969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.353543 (+ 1574us) server_negotiation.cc:407] Connection header received
0929 14:51:32.354541 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.354545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.354599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.354701 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:32.355941 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.356564 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.357723 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.357955 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.358089 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.361562 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:32.361570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:32.361576 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:32.361628 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:32.363813 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.364521 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.364527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.364532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.364637 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.364987 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.364993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.364997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.365302 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:32.365414 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.366036 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.366237 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:51:32.367213 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.352591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.98.1:34835)
0929 14:51:32.353354 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.353403 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.354325 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.354888 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.354894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.355293 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:32.355790 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.355805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.356697 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.356705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.357597 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.357606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.358145 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.358150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.358311 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.359132 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:32.359157 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:32.361320 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:32.363990 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.363997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.364017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.364408 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.364741 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.364744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.364745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.364875 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.365479 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:32.365484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:32.365752 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.365922 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.366226 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":214,"threads_started":1}
I20260929 14:51:32.369204 17611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43655
I20260929 14:51:32.369529 17611 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:32.370244 17611 heartbeater.cc:511] Master 127.0.98.62:43655 requested a full tablet report, sending...
I20260929 14:51:32.372677 17420 ts_manager.cc:194] Registered new tserver with Master: 654addf9b4c4456c9f7284510db8fb7e (127.0.98.1:37071)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:51:32.374943 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:51:32.396291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:32.510782 17616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:32.511155 17616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:32.511247 17616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:32.515574 17616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:32.515689 17616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:32.515781 17616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:32.521562 17616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:32.522717 17616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:32.523840 17616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:32.534307 17622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:32.534312 17624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:32.534492 17616 server_base.cc:1034] running on GCE node
W20260929 14:51:32.534307 17621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:32.535054 17616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:32.535715 17616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:32.536919 17616 hybrid_clock.cc:648] HybridClock initialized: now 1790693492536846 us; error 96 us; skew 500 ppm
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:32.539912 17616 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:51:32.540969 17616 webserver.cc:460] Webserver started at http://127.0.98.2:45773/ using document root <none> and password file <none>
I20260929 14:51:32.541585 17616 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:32.541676 17616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:32.541926 17616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:32.544013 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "577b4ed133e948db91e578d7925f389e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "797589df8e4411a487cfca3bada86bca"
server_key_iv: "4d194e3fcc3ec09d2f90b2c06ce30603"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.544682 17616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "577b4ed133e948db91e578d7925f389e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "797589df8e4411a487cfca3bada86bca"
server_key_iv: "4d194e3fcc3ec09d2f90b2c06ce30603"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.549734 17616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 14:51:32.554441 17631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.556545 17616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:32.556720 17616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "577b4ed133e948db91e578d7925f389e"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "797589df8e4411a487cfca3bada86bca"
server_key_iv: "4d194e3fcc3ec09d2f90b2c06ce30603"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.556887 17616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:32.586014 17616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:32.589557 17616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:32.589748 17616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:32.590422 17616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:32.591410 17616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:32.591485 17616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.591558 17616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:32.591605 17616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.602627 17616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43775
I20260929 14:51:32.602670 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43775 every 8 connection(s)
I20260929 14:51:32.603762 17616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:32.605087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17616
I20260929 14:51:32.605195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:32.605535 392 external_mini_cluster.cc:1428] Setting key 535fa3f5a46e3b8eade5e011878241e0
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:51:32.620673 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.606073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42915 (local address 127.0.98.62:43655)
0929 14:51:32.606297 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.606301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.607284 (+ 983us) server_negotiation.cc:407] Connection header received
0929 14:51:32.608397 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.608403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.608476 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.608584 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:32.610028 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.610691 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.612583 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.612822 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.612935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.616227 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:32.616231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:32.616234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:32.616259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:32.617844 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.618443 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.618448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.618450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.618617 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.619127 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.619136 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.619137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.619399 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:32.619516 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.620358 (+ 842us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.620525 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:51:32.622187 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.606554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.98.2:42915)
0929 14:51:32.607077 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.607143 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.608138 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.608761 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.608767 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.609132 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:32.609855 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.609873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.610834 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.610840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.612426 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.612440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.612994 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.613002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.613231 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.613966 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:32.613993 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:32.615980 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:32.617992 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.617999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.618020 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.618330 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.618745 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.618751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.618755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.619002 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.619517 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:32.619528 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:32.619916 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.620210 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.620696 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":150,"threads_started":1}
I20260929 14:51:32.623750 17745 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43655
I20260929 14:51:32.624090 17745 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:32.624820 17745 heartbeater.cc:511] Master 127.0.98.62:43655 requested a full tablet report, sending...
I20260929 14:51:32.626245 17420 ts_manager.cc:194] Registered new tserver with Master: 577b4ed133e948db91e578d7925f389e (127.0.98.2:43775)
I20260929 14:51:32.626950 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:51:32.640812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:32.753433 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:32.753695 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:32.753736 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:32.757735 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:32.757797 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:32.757853 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:32.763077 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:32.764104 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:32.765133 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:32.774876 17758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:32.774932 17755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:32.774986 17756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:32.775182 17750 server_base.cc:1034] running on GCE node
I20260929 14:51:32.775718 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:32.776374 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:32.777566 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790693492777542 us; error 52 us; skew 500 ppm
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:32.780642 17750 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:51:32.781551 17750 webserver.cc:460] Webserver started at http://127.0.98.3:42003/ using document root <none> and password file <none>
I20260929 14:51:32.782145 17750 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:32.782212 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:32.782418 17750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:32.784474 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f75cd7667ff4058b9852ff8a20e156a"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "0434bada18f0c8b8627b015623a0ec3e"
server_key_iv: "edb9c07365c73a7c2034117a23ec7329"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.785022 17750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f75cd7667ff4058b9852ff8a20e156a"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "0434bada18f0c8b8627b015623a0ec3e"
server_key_iv: "edb9c07365c73a7c2034117a23ec7329"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.789034 17750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 14:51:32.791651 17765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.793200 17750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:51:32.793324 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "6f75cd7667ff4058b9852ff8a20e156a"
format_stamp: "Formatted at 2026-09-29 14:51:32 on dist-test-slave-5hd1"
server_key: "0434bada18f0c8b8627b015623a0ec3e"
server_key_iv: "edb9c07365c73a7c2034117a23ec7329"
server_key_version: "encryptionkey@0"
I20260929 14:51:32.793431 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:32.819552 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:32.822835 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:32.823058 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:32.823711 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:32.824661 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:32.824739 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.824819 17750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:32.824872 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:32.835897 17750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45723
I20260929 14:51:32.835991 17878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45723 every 8 connection(s)
I20260929 14:51:32.837074 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:32.839141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17750
I20260929 14:51:32.839301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:32.839664 392 external_mini_cluster.cc:1428] Setting key 2e1e90f032dae29248512b7c098ac614
Sep 29 14:51:32 dist-test-slave-5hd1 krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:32.852625 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.839348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56403 (local address 127.0.98.62:43655)
0929 14:51:32.839488 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.839493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.840708 (+ 1215us) server_negotiation.cc:407] Connection header received
0929 14:51:32.842071 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.842074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.842135 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.842242 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:32.843692 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.844383 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.845731 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.845952 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.846058 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.849042 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:32.849045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:32.849048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:32.849070 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:32.850489 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.851050 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.851053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.851054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.851147 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.851505 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.851507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.851508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.851693 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:32.851795 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.852364 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.852475 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:51:32.853955 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.839803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43655 (local address 127.0.98.3:56403)
0929 14:51:32.840491 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.840563 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.841783 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.842448 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.842456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.842884 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:32.843485 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.843500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.844545 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.844549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.845586 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.845599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.846126 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.846129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.846304 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.847070 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:32.847088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:32.848839 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:32.850609 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.850615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.850629 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.850935 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.851258 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:32.851260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:32.851261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:32.851401 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:32.851812 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:32.851818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:32.852099 (+ 281us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.852329 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.852764 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":179,"threads_started":1}
I20260929 14:51:32.855235 17879 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43655
I20260929 14:51:32.855531 17879 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:32.856127 17879 heartbeater.cc:511] Master 127.0.98.62:43655 requested a full tablet report, sending...
I20260929 14:51:32.857355 17420 ts_manager.cc:194] Registered new tserver with Master: 6f75cd7667ff4058b9852ff8a20e156a (127.0.98.3:45723)
I20260929 14:51:32.858031 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56403
I20260929 14:51:32.866344 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:32.877885 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.868780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.0.98.62:43655)
0929 14:51:32.868960 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.868965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.869185 (+ 220us) server_negotiation.cc:407] Connection header received
0929 14:51:32.869307 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.869309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.869363 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.869431 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:32.870170 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.870731 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.871707 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.872047 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.872159 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.873763 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:32.873766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:32.873768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:32.873815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:32.875561 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.875994 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.875998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.875999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.876089 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:32.876665 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:32.876668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:32.876669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:32.877038 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:32.877211 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.877495 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.877690 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:51:32.881305 17420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:32.885124 17420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:32.901397 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.894785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45723 (local address 127.0.0.1:55224)
0929 14:51:32.895207 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.895278 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.895439 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.896231 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.896238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.896264 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:32.896530 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.896539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.898072 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.898077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.899482 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.899492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.900573 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.900577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.900818 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.900917 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.901061 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.901203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":98,"thread_start_us":105,"threads_started":1}
I20260929 14:51:32.901486 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.895002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.0.98.3:45723)
0929 14:51:32.895598 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.895614 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.895645 (+ 31us) server_negotiation.cc:407] Connection header received
0929 14:51:32.895745 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.895753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.895999 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.896244 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:32.896677 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.897940 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.899632 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.900453 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.900574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.900791 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.901115 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.901279 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":191,"threads_started":1}
I20260929 14:51:32.902799 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.896535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.98.1:37071)
0929 14:51:32.897015 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.897020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.897041 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:51:32.897104 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.897108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.897317 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.897781 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:32.897910 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.899064 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.901008 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.901911 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.902293 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.902453 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.902507 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.902625 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":109,"threads_started":1}
I20260929 14:51:32.902894 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.895830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.0.98.2:43775)
0929 14:51:32.896727 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:32.896744 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:32.896770 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:51:32.896850 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:32.896856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:32.897059 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:32.897320 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:32.897661 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.899025 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.901042 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.901887 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.902043 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.902213 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:32.902621 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 14:51:32.902729 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":128,"threads_started":1}
I20260929 14:51:32.903407 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.895620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43775 (local address 127.0.0.1:54154)
0929 14:51:32.895891 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.895910 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.896012 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.897263 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.897267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.897289 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:32.897463 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.897483 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.899206 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.899210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.900911 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.900922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.902037 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.902044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.902205 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.902284 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.902357 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.903237 (+ 880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":396,"thread_start_us":109,"threads_started":1}
I20260929 14:51:32.903527 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:32.895968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37071 (local address 127.0.0.1:52494)
0929 14:51:32.896315 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:32.896329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:32.896412 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:32.897485 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:32.897489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:32.897507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:32.897633 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.897639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.899207 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.899210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.900841 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:32.900857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:32.902071 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:32.902075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:32.902230 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:32.902347 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 14:51:32.902430 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 14:51:32.903290 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":912,"thread_start_us":110,"threads_started":1}
I20260929 14:51:32.905033 17813 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a2833072046c4ba06299b3e902d30 (DEFAULT_TABLE table=test-table [id=182d38e8c7214623bd79e75d21b13681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:32.906203 17813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a2833072046c4ba06299b3e902d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:32.906183 17545 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a2833072046c4ba06299b3e902d30 (DEFAULT_TABLE table=test-table [id=182d38e8c7214623bd79e75d21b13681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:32.906165 17679 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a2833072046c4ba06299b3e902d30 (DEFAULT_TABLE table=test-table [id=182d38e8c7214623bd79e75d21b13681]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:32.907150 17545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a2833072046c4ba06299b3e902d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:32.907401 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a2833072046c4ba06299b3e902d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:32.914207 17896 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: Bootstrap starting.
I20260929 14:51:32.914984 17897 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Bootstrap starting.
I20260929 14:51:32.916579 17898 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Bootstrap starting.
I20260929 14:51:32.917296 17896 tablet_bootstrap.cc:654] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:32.917328 17897 tablet_bootstrap.cc:654] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:32.918183 17896 log.cc:824] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:32.918560 17897 log.cc:824] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:32.919046 17898 tablet_bootstrap.cc:654] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:32.919939 17898 log.cc:824] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:32.920441 17896 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: No bootstrap required, opened a new log
I20260929 14:51:32.920677 17896 ts_tablet_manager.cc:1397] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:51:32.920928 17897 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: No bootstrap required, opened a new log
I20260929 14:51:32.921118 17897 ts_tablet_manager.cc:1397] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:32.922608 17898 tablet_bootstrap.cc:492] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: No bootstrap required, opened a new log
I20260929 14:51:32.922798 17898 ts_tablet_manager.cc:1397] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:32.925053 17897 raft_consensus.cc:348] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.925091 17896 raft_consensus.cc:348] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.925343 17897 raft_consensus.cc:374] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:32.925369 17896 raft_consensus.cc:374] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:32.925429 17896 raft_consensus.cc:729] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654addf9b4c4456c9f7284510db8fb7e, State: Initialized, Role: FOLLOWER
I20260929 14:51:32.925442 17897 raft_consensus.cc:729] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f75cd7667ff4058b9852ff8a20e156a, State: Initialized, Role: FOLLOWER
I20260929 14:51:32.925925 17897 consensus_queue.cc:260] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.926012 17896 consensus_queue.cc:260] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.926764 17879 heartbeater.cc:503] Master 127.0.98.62:43655 was elected leader, sending a full tablet report...
I20260929 14:51:32.926967 17897 ts_tablet_manager.cc:1428] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:51:32.926877 17898 raft_consensus.cc:348] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.927135 17898 raft_consensus.cc:374] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:32.927184 17898 raft_consensus.cc:729] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577b4ed133e948db91e578d7925f389e, State: Initialized, Role: FOLLOWER
I20260929 14:51:32.927202 17896 ts_tablet_manager.cc:1428] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 14:51:32.927520 17611 heartbeater.cc:503] Master 127.0.98.62:43655 was elected leader, sending a full tablet report...
I20260929 14:51:32.927657 17898 consensus_queue.cc:260] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:32.928457 17745 heartbeater.cc:503] Master 127.0.98.62:43655 was elected leader, sending a full tablet report...
I20260929 14:51:32.929878 17898 ts_tablet_manager.cc:1428] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:51:33.014202 17904 raft_consensus.cc:482] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:33.014467 17904 raft_consensus.cc:504] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:33.015704 17904 leader_election.cc:288] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f75cd7667ff4058b9852ff8a20e156a (127.0.98.3:45723), 654addf9b4c4456c9f7284510db8fb7e (127.0.98.1:37071)
I20260929 14:51:33.020282 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.016345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45723 (local address 127.0.98.2:52929)
0929 14:51:33.016538 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.016562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:33.016709 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:33.017139 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:33.017143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:33.017165 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:33.017321 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.017328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.018117 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.018123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.018981 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.018990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.019753 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.019756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.019878 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.019912 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 14:51:33.020012 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 14:51:33.020098 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 14:51:33.020304 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.016447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52929 (local address 127.0.98.3:45723)
0929 14:51:33.016641 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.016647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.016666 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:51:33.016912 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.016919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.017008 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.017106 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:33.017439 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.017994 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.019112 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.019594 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.019701 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.019806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.020031 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.020106 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:51:33.021077 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.016698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37071 (local address 127.0.98.2:52321)
0929 14:51:33.017116 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.017134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:33.017274 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:33.017582 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:33.017586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:33.017603 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:33.017762 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.017768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.018590 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.018597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.019517 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.019550 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.020595 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.020601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.020713 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.020735 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:51:33.020826 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 14:51:33.020911 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":190,"threads_started":1}
I20260929 14:51:33.021152 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.016920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52321 (local address 127.0.98.1:37071)
0929 14:51:33.017108 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.017113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.017231 (+ 118us) server_negotiation.cc:407] Connection header received
0929 14:51:33.017401 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.017405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.017460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.017559 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:33.017880 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.018460 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.019676 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.020379 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.020502 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.020586 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.020900 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.021021 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 14:51:33.021291 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a2833072046c4ba06299b3e902d30" candidate_uuid: "577b4ed133e948db91e578d7925f389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75cd7667ff4058b9852ff8a20e156a" is_pre_election: true
I20260929 14:51:33.021736 17833 raft_consensus.cc:2454] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 577b4ed133e948db91e578d7925f389e in term 0.
I20260929 14:51:33.021920 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a2833072046c4ba06299b3e902d30" candidate_uuid: "577b4ed133e948db91e578d7925f389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654addf9b4c4456c9f7284510db8fb7e" is_pre_election: true
I20260929 14:51:33.022289 17565 raft_consensus.cc:2454] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 577b4ed133e948db91e578d7925f389e in term 0.
I20260929 14:51:33.022398 17632 leader_election.cc:302] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577b4ed133e948db91e578d7925f389e, 6f75cd7667ff4058b9852ff8a20e156a; no voters:
I20260929 14:51:33.023048 17904 raft_consensus.cc:2781] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:33.023137 17904 raft_consensus.cc:482] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:33.023170 17904 raft_consensus.cc:3037] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:33.024577 17904 raft_consensus.cc:504] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:33.024994 17904 leader_election.cc:288] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [CANDIDATE]: Term 1 election: Requested vote from peers 6f75cd7667ff4058b9852ff8a20e156a (127.0.98.3:45723), 654addf9b4c4456c9f7284510db8fb7e (127.0.98.1:37071)
I20260929 14:51:33.025460 17565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a2833072046c4ba06299b3e902d30" candidate_uuid: "577b4ed133e948db91e578d7925f389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654addf9b4c4456c9f7284510db8fb7e"
I20260929 14:51:33.025601 17565 raft_consensus.cc:3037] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:33.025652 17833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a2833072046c4ba06299b3e902d30" candidate_uuid: "577b4ed133e948db91e578d7925f389e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f75cd7667ff4058b9852ff8a20e156a"
I20260929 14:51:33.025892 17833 raft_consensus.cc:3037] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:33.026831 17565 raft_consensus.cc:2454] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577b4ed133e948db91e578d7925f389e in term 1.
I20260929 14:51:33.027227 17632 leader_election.cc:302] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577b4ed133e948db91e578d7925f389e, 654addf9b4c4456c9f7284510db8fb7e; no voters:
I20260929 14:51:33.027433 17833 raft_consensus.cc:2454] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577b4ed133e948db91e578d7925f389e in term 1.
I20260929 14:51:33.027488 17904 raft_consensus.cc:2781] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:33.027750 17904 raft_consensus.cc:686] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 1 LEADER]: Becoming Leader. State: Replica: 577b4ed133e948db91e578d7925f389e, State: Running, Role: LEADER
I20260929 14:51:33.028292 17904 consensus_queue.cc:237] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } }
I20260929 14:51:33.032307 17420 catalog_manager.cc:5495] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e reported cstate change: term changed from 0 to 1, leader changed from <none> to 577b4ed133e948db91e578d7925f389e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "577b4ed133e948db91e578d7925f389e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577b4ed133e948db91e578d7925f389e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 } health_report { overall_health: UNKNOWN } } }
I20260929 14:51:33.061764 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.057889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.2:43775)
0929 14:51:33.058091 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.058095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.058108 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:51:33.058191 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.058193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.058246 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.058361 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:33.058618 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.059117 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.060040 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.060335 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.060469 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.060889 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:33.061344 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:33.061442 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.061556 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.061607 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:51:33.068965 17565 raft_consensus.cc:1261] T ec1a2833072046c4ba06299b3e902d30 P 654addf9b4c4456c9f7284510db8fb7e [term 1 FOLLOWER]: Refusing update from remote peer 577b4ed133e948db91e578d7925f389e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:33.069090 17833 raft_consensus.cc:1261] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 1 FOLLOWER]: Refusing update from remote peer 577b4ed133e948db91e578d7925f389e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:33.070011 17904 consensus_queue.cc:1035] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [LEADER]: Connected to new peer: Peer: permanent_uuid: "654addf9b4c4456c9f7284510db8fb7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:33.070313 17906 consensus_queue.cc:1035] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f75cd7667ff4058b9852ff8a20e156a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:33.089812 17420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:33.090021 17420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:33.093152 17420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30f3cabc38e34e01b398b99eb2b7cccc: Sending DeleteTablet for 3 replicas of tablet ec1a2833072046c4ba06299b3e902d30
I20260929 14:51:33.094249 17813 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a2833072046c4ba06299b3e902d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55224
I20260929 14:51:33.094379 17545 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a2833072046c4ba06299b3e902d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52494
I20260929 14:51:33.094875 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a2833072046c4ba06299b3e902d30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54154
I20260929 14:51:33.095175 17920 tablet_replica.cc:330] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: stopping tablet replica
I20260929 14:51:33.095353 17921 tablet_replica.cc:330] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: stopping tablet replica
I20260929 14:51:33.095497 17920 raft_consensus.cc:2229] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:33.095785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17482
I20260929 14:51:33.095837 17920 raft_consensus.cc:2258] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:33.097517 17920 ts_tablet_manager.cc:1905] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:33.099010 17921 raft_consensus.cc:2229] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:51:33.099407 17921 raft_consensus.cc:2258] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:33.100896 17920 ts_tablet_manager.cc:1918] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:33.101001 17920 log.cc:1196] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/ec1a2833072046c4ba06299b3e902d30
I20260929 14:51:33.101337 17920 ts_tablet_manager.cc:1939] T ec1a2833072046c4ba06299b3e902d30 P 6f75cd7667ff4058b9852ff8a20e156a: Deleting consensus metadata
I20260929 14:51:33.101470 17921 ts_tablet_manager.cc:1905] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:33.102658 17404 catalog_manager.cc:4841] TS 6f75cd7667ff4058b9852ff8a20e156a (127.0.98.3:45723): tablet ec1a2833072046c4ba06299b3e902d30 (table test-table [id=182d38e8c7214623bd79e75d21b13681]) successfully deleted
W20260929 14:51:33.104916 17404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37071 (error 108)
I20260929 14:51:33.104939 17921 ts_tablet_manager.cc:1918] T ec1a2833072046c4ba06299b3e902d30 P 577b4ed133e948db91e578d7925f389e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:33.105113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17616
I20260929 14:51:33.106868 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.106452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37071 (local address 127.0.0.1:52504)
0929 14:51:33.106652 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.106747 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 14:51:33.107036 17404 catalog_manager.cc:4584] TS 654addf9b4c4456c9f7284510db8fb7e (127.0.98.1:37071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec1a2833072046c4ba06299b3e902d30: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37071: connect: Connection refused (error 111)
I20260929 14:51:33.112267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17750
I20260929 14:51:33.114846 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.114486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43775 (local address 127.0.0.1:54172)
0929 14:51:33.114662 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.114738 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 14:51:33.119115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17388
2026-09-29T14:51:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2477 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T14:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:33Z Disabled control of system clock
I20260929 14:51:33.154922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43017
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:33.266404 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:33.266675 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:33.266711 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:33.270772 17926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:33.270831 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:33.270846 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:33.270864 17926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:33.270879 17926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:33.275560 17926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:33.276674 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:33.278039 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:33.286928 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.286945 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.286931 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:33.287312 17926 server_base.cc:1034] running on GCE node
I20260929 14:51:33.287983 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:33.289136 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:33.290341 17926 hybrid_clock.cc:648] HybridClock initialized: now 1790693493290326 us; error 52 us; skew 500 ppm
I20260929 14:51:33.292722 17926 webserver.cc:460] Webserver started at http://127.0.98.62:45977/ using document root <none> and password file <none>
I20260929 14:51:33.293311 17926 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:33.293402 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:33.293608 17926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:33.295620 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "6a8799cbf0294142a243cf49906e0467"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "c1086cf00511736b6dde8d7acc4ac02f"
server_key_iv: "1d35bb3434e88699b964987a14869dba"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.296223 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a8799cbf0294142a243cf49906e0467"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "c1086cf00511736b6dde8d7acc4ac02f"
server_key_iv: "1d35bb3434e88699b964987a14869dba"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.299923 17926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:33.302642 17940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.304116 17926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 14:51:33.304304 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "6a8799cbf0294142a243cf49906e0467"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "c1086cf00511736b6dde8d7acc4ac02f"
server_key_iv: "1d35bb3434e88699b964987a14869dba"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.304430 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:33.319116 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:33.319938 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:33.320148 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:33.328312 17926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43017
I20260929 14:51:33.328425 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43017 every 8 connection(s)
I20260929 14:51:33.329393 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:33.331669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 17926
I20260929 14:51:33.331818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:33.332154 392 external_mini_cluster.cc:1428] Setting key eb2246da2f3b594147f4a750e660ea05
I20260929 14:51:33.334084 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:33.342146 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: Bootstrap starting.
I20260929 14:51:33.342924 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.333974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.0.98.62:43017)
0929 14:51:33.334838 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.334856 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.334897 (+ 41us) server_negotiation.cc:407] Connection header received
0929 14:51:33.335677 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.335695 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.336149 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.336590 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:33.337046 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.338132 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.339068 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.339595 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.339778 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.340854 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:33.340859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:33.340872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:33.341313 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:33.341375 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:33.341409 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:33.341529 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.341707 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.342004 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":221,"threads_started":1}
I20260929 14:51:33.345263 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:33.346320 17993 log.cc:824] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:33.349144 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: No bootstrap required, opened a new log
I20260929 14:51:33.353641 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } }
I20260929 14:51:33.354013 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:33.354108 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8799cbf0294142a243cf49906e0467, State: Initialized, Role: FOLLOWER
I20260929 14:51:33.354638 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } }
I20260929 14:51:33.354781 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:33.354840 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:33.354928 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:33.356362 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } }
I20260929 14:51:33.356767 17993 leader_election.cc:302] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a8799cbf0294142a243cf49906e0467; no voters:
I20260929 14:51:33.357111 17993 leader_election.cc:288] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:33.357280 17998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:33.357591 17998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [term 1 LEADER]: Becoming Leader. State: Replica: 6a8799cbf0294142a243cf49906e0467, State: Running, Role: LEADER
I20260929 14:51:33.357999 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } }
I20260929 14:51:33.358851 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:33.359876 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a8799cbf0294142a243cf49906e0467. Latest consensus state: current_term: 1 leader_uuid: "6a8799cbf0294142a243cf49906e0467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } } }
I20260929 14:51:33.360021 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:33.360226 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a8799cbf0294142a243cf49906e0467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8799cbf0294142a243cf49906e0467" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43017 } } }
I20260929 14:51:33.360361 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:33.360538 18006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:33.363508 18006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:33.369930 18006 catalog_manager.cc:1347] Generated new cluster ID: bfd8551ef916499e945b54430de6e692
I20260929 14:51:33.370020 18006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:33.384703 18006 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:33.385962 18006 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:33.394012 18006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: Generated new TSK 0
I20260929 14:51:33.395192 18006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:33.407783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--builtin_ntp_servers=127.0.98.20:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:33.525911 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:33.526233 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:33.526278 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:33.530608 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:33.530676 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:33.530733 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:33.535583 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:33.536849 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:33.538244 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:33.548668 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.548679 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.548687 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:33.549155 18017 server_base.cc:1034] running on GCE node
I20260929 14:51:33.549891 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:33.550936 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:33.552242 18017 hybrid_clock.cc:648] HybridClock initialized: now 1790693493552139 us; error 56 us; skew 500 ppm
I20260929 14:51:33.554844 18017 webserver.cc:460] Webserver started at http://127.0.98.1:39333/ using document root <none> and password file <none>
I20260929 14:51:33.555666 18017 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:33.555774 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:33.556043 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:33.558707 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "b06b0fd0792c471cb96b658ff77db280"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "fa2bca6c81c651e1bf6f1281eaf98c36"
server_key_iv: "f53513646aca7f1fb633c2711f3d302a"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.559512 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b06b0fd0792c471cb96b658ff77db280"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "fa2bca6c81c651e1bf6f1281eaf98c36"
server_key_iv: "f53513646aca7f1fb633c2711f3d302a"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.564549 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:51:33.567855 18031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.569680 18017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:33.569971 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "b06b0fd0792c471cb96b658ff77db280"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "fa2bca6c81c651e1bf6f1281eaf98c36"
server_key_iv: "f53513646aca7f1fb633c2711f3d302a"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.570153 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:33.594982 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:33.595934 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:33.596222 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:33.596987 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:33.598208 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:33.598285 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.598359 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:33.598402 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.610062 18017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34619
I20260929 14:51:33.610142 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34619 every 8 connection(s)
I20260929 14:51:33.611268 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:33.615618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18017
I20260929 14:51:33.615742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:33.616060 392 external_mini_cluster.cc:1428] Setting key d001e046abec7bcb954538abc0d3a61c
I20260929 14:51:33.618563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--builtin_ntp_servers=127.0.98.20:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:33.623133 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.613539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39693 (local address 127.0.98.62:43017)
0929 14:51:33.613696 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.613700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.615195 (+ 1495us) server_negotiation.cc:407] Connection header received
0929 14:51:33.617185 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.617188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.617251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.617350 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:33.618589 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.619275 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.620777 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.621024 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.621177 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.622615 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:33.622618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:33.622620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:33.622631 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:33.622648 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:33.622655 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:33.622732 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.622895 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.622980 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":69}
I20260929 14:51:33.623936 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.614187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43017 (local address 127.0.98.1:39693)
0929 14:51:33.614990 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.615043 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:33.616946 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:33.617532 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:33.617539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:33.617657 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:33.618412 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.618435 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.619429 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.619435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.620649 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.620659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.621303 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.621309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.621465 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.622339 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:33.622341 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:33.622379 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:33.622440 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:33.622747 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:33.622757 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:33.622763 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:51:33.622883 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 14:51:33.623184 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":154,"threads_started":1}
I20260929 14:51:33.625905 18145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43017
I20260929 14:51:33.626154 18145 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:33.626667 18145 heartbeater.cc:511] Master 127.0.98.62:43017 requested a full tablet report, sending...
I20260929 14:51:33.628881 17954 ts_manager.cc:194] Registered new tserver with Master: b06b0fd0792c471cb96b658ff77db280 (127.0.98.1:34619)
I20260929 14:51:33.631263 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39693
W20260929 14:51:33.762192 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:33.762547 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:33.762593 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:33.767083 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:33.767163 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:33.767228 18148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:33.772637 18148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:33.774215 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:33.776082 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:33.788206 18156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.788218 18154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.788206 18153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:33.788621 18148 server_base.cc:1034] running on GCE node
I20260929 14:51:33.789162 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:33.789942 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:33.791177 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790693493791164 us; error 49 us; skew 500 ppm
I20260929 14:51:33.793375 18148 webserver.cc:460] Webserver started at http://127.0.98.2:42971/ using document root <none> and password file <none>
I20260929 14:51:33.794041 18148 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:33.794134 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:33.794368 18148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:33.796559 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "c750b2a518d34c6bb2d301627235112d"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "b27cb6e64bf250129123db86c051d0ba"
server_key_iv: "7e47b130b99f481d8bb410d08de10138"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.797132 18148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c750b2a518d34c6bb2d301627235112d"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "b27cb6e64bf250129123db86c051d0ba"
server_key_iv: "7e47b130b99f481d8bb410d08de10138"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.801173 18148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:51:33.803879 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.805404 18148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:33.805552 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "c750b2a518d34c6bb2d301627235112d"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "b27cb6e64bf250129123db86c051d0ba"
server_key_iv: "7e47b130b99f481d8bb410d08de10138"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.805665 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:33.817077 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:33.817718 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:33.817906 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:33.818471 18148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:33.819411 18148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:33.819482 18148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.819561 18148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:33.819603 18148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.829669 18148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45747
I20260929 14:51:33.829749 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45747 every 8 connection(s)
I20260929 14:51:33.830595 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:33.838124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18148
I20260929 14:51:33.838316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:33.838640 392 external_mini_cluster.cc:1428] Setting key 98569ccc61d87a38bb09f1acea7bfa90
I20260929 14:51:33.841365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--builtin_ntp_servers=127.0.98.20:40169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:33.841552 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.832724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54549 (local address 127.0.98.62:43017)
0929 14:51:33.832922 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:33.832926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:33.834002 (+ 1076us) server_negotiation.cc:407] Connection header received
0929 14:51:33.835533 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:33.835535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:33.835589 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:33.835673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:33.836846 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.837396 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.838913 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.839214 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.839317 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.840956 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:33.840960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:33.840963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:33.840976 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:33.841036 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:33.841047 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:33.841141 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:33.841317 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 14:51:33.841385 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 14:51:33.842885 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:33.833043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43017 (local address 127.0.98.2:54549)
0929 14:51:33.833809 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:33.833858 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:33.835313 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:33.835898 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:33.835905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:33.836005 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:33.836674 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.836688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.837563 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.837567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.838747 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:33.838771 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:33.839422 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:33.839430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:33.839647 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:33.840601 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:33.840603 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:33.840669 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:33.840779 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:33.841151 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:33.841159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:33.841163 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:51:33.841288 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 14:51:33.841653 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260929 14:51:33.844456 18276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43017
I20260929 14:51:33.844790 18276 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:33.845451 18276 heartbeater.cc:511] Master 127.0.98.62:43017 requested a full tablet report, sending...
I20260929 14:51:33.846710 17954 ts_manager.cc:194] Registered new tserver with Master: c750b2a518d34c6bb2d301627235112d (127.0.98.2:45747)
I20260929 14:51:33.847419 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54549
W20260929 14:51:33.957371 18279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:33.957648 18279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:33.957687 18279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:33.962023 18279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:33.962082 18279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:33.962136 18279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:33.966830 18279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:33.967893 18279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:33.969329 18279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:33.979684 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:33.979749 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:33.979884 18279 server_base.cc:1034] running on GCE node
W20260929 14:51:33.980126 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:33.980635 18279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:33.981315 18279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:33.982547 18279 hybrid_clock.cc:648] HybridClock initialized: now 1790693493982508 us; error 62 us; skew 500 ppm
I20260929 14:51:33.985141 18279 webserver.cc:460] Webserver started at http://127.0.98.3:34437/ using document root <none> and password file <none>
I20260929 14:51:33.985795 18279 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:33.985903 18279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:33.986440 18279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:33.989032 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb8be85d17c8453d8fc369a0b6ab2809"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "e6bbc454836e92ebe3012db81a02962f"
server_key_iv: "8b911f1d74af0d1d166a9e5117e91b75"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.989629 18279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb8be85d17c8453d8fc369a0b6ab2809"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "e6bbc454836e92ebe3012db81a02962f"
server_key_iv: "8b911f1d74af0d1d166a9e5117e91b75"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.993731 18279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:51:33.996572 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:33.997987 18279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:33.998162 18279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "cb8be85d17c8453d8fc369a0b6ab2809"
format_stamp: "Formatted at 2026-09-29 14:51:33 on dist-test-slave-5hd1"
server_key: "e6bbc454836e92ebe3012db81a02962f"
server_key_iv: "8b911f1d74af0d1d166a9e5117e91b75"
server_key_version: "encryptionkey@0"
I20260929 14:51:33.998281 18279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:34.027077 18279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:34.027851 18279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:34.028079 18279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:34.028726 18279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:34.029735 18279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:34.029815 18279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:34.029884 18279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:34.029928 18279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:34.040315 18279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43915
I20260929 14:51:34.040481 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43915 every 8 connection(s)
I20260929 14:51:34.041355 18279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:34.049602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18279
I20260929 14:51:34.049746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:34.050088 392 external_mini_cluster.cc:1428] Setting key cc91ee7ea944b8c1c92b07923028bc05
I20260929 14:51:34.053431 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.043902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34015 (local address 127.0.98.62:43017)
0929 14:51:34.044103 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.044109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.045480 (+ 1371us) server_negotiation.cc:407] Connection header received
0929 14:51:34.047116 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.047119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.047191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.047304 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:34.048560 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.049108 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.050806 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.051085 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.051199 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.052824 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:34.052826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:34.052829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:34.052841 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:34.052862 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:34.052870 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:34.052936 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.053218 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.053302 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 14:51:34.054528 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.044376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43017 (local address 127.0.98.3:34015)
0929 14:51:34.045270 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.045326 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.046851 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.047516 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.047524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.047625 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:34.048369 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.048387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.049292 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.049297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.050673 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.050683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.051381 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.051385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.051586 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.052498 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:34.052502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:34.052561 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:34.052638 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:34.052985 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:34.052993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:34.052999 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.053132 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.053464 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":141,"threads_started":1}
I20260929 14:51:34.055902 18407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43017
I20260929 14:51:34.056274 18407 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:34.057039 18407 heartbeater.cc:511] Master 127.0.98.62:43017 requested a full tablet report, sending...
I20260929 14:51:34.058337 17954 ts_manager.cc:194] Registered new tserver with Master: cb8be85d17c8453d8fc369a0b6ab2809 (127.0.98.3:43915)
I20260929 14:51:34.059095 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34015
I20260929 14:51:34.064837 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:34.071224 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.067138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.0.98.62:43017)
0929 14:51:34.067297 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.067301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.067615 (+ 314us) server_negotiation.cc:407] Connection header received
0929 14:51:34.067673 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.067675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.067723 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.067839 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:34.068117 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.068748 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.069623 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.069874 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.069971 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.070804 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:34.070807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:34.070809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:34.070822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:34.070843 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:34.070851 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:34.070921 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.071034 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.071090 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:51:34.074590 17954 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:34.078369 17954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:34.094720 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.088070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43915 (local address 127.0.0.1:37546)
0929 14:51:34.089168 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.089219 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.089376 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.090348 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.090352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.090388 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:34.090662 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.090670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.091830 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.091837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.093071 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.093082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.094147 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.094151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.094284 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.094334 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.094459 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.094586 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":55,"thread_start_us":132,"threads_started":1}
I20260929 14:51:34.094866 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.088643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.98.3:43915)
0929 14:51:34.089229 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.089245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.089312 (+ 67us) server_negotiation.cc:407] Connection header received
0929 14:51:34.089677 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.089684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.089960 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.090149 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:34.090800 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.091691 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.093205 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.093940 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.094056 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.094291 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.094523 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.094649 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":120,"threads_started":1}
I20260929 14:51:34.095892 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.088860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45747 (local address 127.0.0.1:49468)
0929 14:51:34.089687 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.089700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.089767 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.090278 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.090283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.090307 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:34.090535 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.090543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.092127 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.092131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.093739 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.093749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.095305 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.095308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.095577 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.095600 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.095682 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.095758 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":253,"thread_start_us":89,"threads_started":1}
I20260929 14:51:34.096504 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.089288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.98.2:45747)
0929 14:51:34.089804 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.089815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.089839 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:51:34.089930 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.089935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.090103 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.090244 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:34.090693 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.091983 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.093901 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.094743 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.094868 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.095063 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.096150 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.096345 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":146,"threads_started":1}
I20260929 14:51:34.096892 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.088341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34619 (local address 127.0.0.1:57358)
0929 14:51:34.089628 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.089642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.089742 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.090414 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.090420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.090437 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:34.090662 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.090670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.092127 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.092131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.093738 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.093749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.096391 (+ 2642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.096395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.096512 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.096535 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.096635 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.096735 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":201,"thread_start_us":147,"threads_started":1}
I20260929 14:51:34.097375 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.088540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.0.98.1:34619)
0929 14:51:34.089549 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.089555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.089873 (+ 318us) server_negotiation.cc:407] Connection header received
0929 14:51:34.089957 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.089961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.090128 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.090299 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:34.090794 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.091983 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.095151 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.095986 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.096098 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.096305 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.097119 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.097228 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":163,"threads_started":1}
I20260929 14:51:34.098663 18341 tablet_service.cc:1467] Processing CreateTablet for tablet aef99e0f87e44babb6f48c17d0a6251c (DEFAULT_TABLE table=test-table [id=2be2f88ecaac48b9ac5bfede1b810379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:34.099845 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef99e0f87e44babb6f48c17d0a6251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:34.099876 18079 tablet_service.cc:1467] Processing CreateTablet for tablet aef99e0f87e44babb6f48c17d0a6251c (DEFAULT_TABLE table=test-table [id=2be2f88ecaac48b9ac5bfede1b810379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:34.100801 18079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef99e0f87e44babb6f48c17d0a6251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:34.102473 18210 tablet_service.cc:1467] Processing CreateTablet for tablet aef99e0f87e44babb6f48c17d0a6251c (DEFAULT_TABLE table=test-table [id=2be2f88ecaac48b9ac5bfede1b810379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:34.103612 18210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef99e0f87e44babb6f48c17d0a6251c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:34.107924 18425 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Bootstrap starting.
I20260929 14:51:34.108496 18424 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Bootstrap starting.
I20260929 14:51:34.111509 18426 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Bootstrap starting.
I20260929 14:51:34.113240 18424 tablet_bootstrap.cc:654] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:34.113732 18425 tablet_bootstrap.cc:654] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:34.113749 18426 tablet_bootstrap.cc:654] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:34.114367 18424 log.cc:824] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:34.114509 18425 log.cc:824] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:34.114504 18426 log.cc:824] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:34.116647 18425 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: No bootstrap required, opened a new log
I20260929 14:51:34.116647 18424 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: No bootstrap required, opened a new log
I20260929 14:51:34.116779 18426 tablet_bootstrap.cc:492] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: No bootstrap required, opened a new log
I20260929 14:51:34.116861 18425 ts_tablet_manager.cc:1397] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 14:51:34.116849 18424 ts_tablet_manager.cc:1397] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260929 14:51:34.117039 18426 ts_tablet_manager.cc:1397] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:34.121011 18426 raft_consensus.cc:348] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.121325 18426 raft_consensus.cc:374] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:34.121384 18426 raft_consensus.cc:729] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c750b2a518d34c6bb2d301627235112d, State: Initialized, Role: FOLLOWER
I20260929 14:51:34.121289 18424 raft_consensus.cc:348] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.121575 18424 raft_consensus.cc:374] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:34.121642 18424 raft_consensus.cc:729] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06b0fd0792c471cb96b658ff77db280, State: Initialized, Role: FOLLOWER
I20260929 14:51:34.121572 18425 raft_consensus.cc:348] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.121820 18425 raft_consensus.cc:374] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:34.121886 18425 raft_consensus.cc:729] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8be85d17c8453d8fc369a0b6ab2809, State: Initialized, Role: FOLLOWER
I20260929 14:51:34.122097 18426 consensus_queue.cc:260] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.122151 18424 consensus_queue.cc:260] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.122239 18425 consensus_queue.cc:260] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.122972 18145 heartbeater.cc:503] Master 127.0.98.62:43017 was elected leader, sending a full tablet report...
I20260929 14:51:34.123206 18425 ts_tablet_manager.cc:1428] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 14:51:34.123502 18407 heartbeater.cc:503] Master 127.0.98.62:43017 was elected leader, sending a full tablet report...
I20260929 14:51:34.123574 18426 ts_tablet_manager.cc:1428] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:51:34.123919 18276 heartbeater.cc:503] Master 127.0.98.62:43017 was elected leader, sending a full tablet report...
I20260929 14:51:34.126317 18424 ts_tablet_manager.cc:1428] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
W20260929 14:51:34.292459 18408 tablet.cc:2369] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:51:34.331804 18277 tablet.cc:2369] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:34.362130 18431 raft_consensus.cc:482] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:34.362661 18431 raft_consensus.cc:504] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
W20260929 14:51:34.362908 18146 tablet.cc:2369] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:34.364233 18431 leader_election.cc:288] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06b0fd0792c471cb96b658ff77db280 (127.0.98.1:34619), c750b2a518d34c6bb2d301627235112d (127.0.98.2:45747)
I20260929 14:51:34.368525 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.364632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45747 (local address 127.0.98.3:53399)
0929 14:51:34.364901 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.364925 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.365045 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.365358 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.365361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.365378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:34.365534 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.365541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.366410 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.366415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.367337 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.367347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.368016 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.368020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.368128 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.368151 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.368275 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.368366 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 14:51:34.368525 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.364809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53399 (local address 127.0.98.2:45747)
0929 14:51:34.364985 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.364991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.365018 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:51:34.365183 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.365185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.365237 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.365333 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:34.365649 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.366294 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.367466 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.367885 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.368010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.368088 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.368285 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.368364 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:51:34.369261 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.364664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34619 (local address 127.0.98.3:43165)
0929 14:51:34.365287 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.365302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:34.365534 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:34.365846 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:34.365849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:34.365865 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:34.366037 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.366045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.366880 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.366887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.367744 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:34.367753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.368537 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.368541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:34.368854 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.368887 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 14:51:34.368984 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:51:34.369065 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":155,"threads_started":1}
I20260929 14:51:34.369319 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.364780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43165 (local address 127.0.98.1:34619)
0929 14:51:34.365041 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.365046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.365410 (+ 364us) server_negotiation.cc:407] Connection header received
0929 14:51:34.365681 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.365689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.365739 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.365829 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:34.366193 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.366756 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.367885 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.368385 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.368517 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.368589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.369044 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.369180 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 14:51:34.369519 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef99e0f87e44babb6f48c17d0a6251c" candidate_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c750b2a518d34c6bb2d301627235112d" is_pre_election: true
I20260929 14:51:34.369979 18230 raft_consensus.cc:2454] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb8be85d17c8453d8fc369a0b6ab2809 in term 0.
I20260929 14:51:34.370155 18099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef99e0f87e44babb6f48c17d0a6251c" candidate_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06b0fd0792c471cb96b658ff77db280" is_pre_election: true
I20260929 14:51:34.370528 18099 raft_consensus.cc:2454] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb8be85d17c8453d8fc369a0b6ab2809 in term 0.
I20260929 14:51:34.370625 18296 leader_election.cc:302] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c750b2a518d34c6bb2d301627235112d, cb8be85d17c8453d8fc369a0b6ab2809; no voters:
I20260929 14:51:34.371052 18431 raft_consensus.cc:2781] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:34.371176 18431 raft_consensus.cc:482] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:34.371243 18431 raft_consensus.cc:3037] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:34.372437 18431 raft_consensus.cc:504] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.372874 18431 leader_election.cc:288] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [CANDIDATE]: Term 1 election: Requested vote from peers b06b0fd0792c471cb96b658ff77db280 (127.0.98.1:34619), c750b2a518d34c6bb2d301627235112d (127.0.98.2:45747)
I20260929 14:51:34.373349 18099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef99e0f87e44babb6f48c17d0a6251c" candidate_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06b0fd0792c471cb96b658ff77db280"
I20260929 14:51:34.373342 18230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef99e0f87e44babb6f48c17d0a6251c" candidate_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c750b2a518d34c6bb2d301627235112d"
I20260929 14:51:34.373498 18099 raft_consensus.cc:3037] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:34.373505 18230 raft_consensus.cc:3037] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:34.374790 18099 raft_consensus.cc:2454] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb8be85d17c8453d8fc369a0b6ab2809 in term 1.
I20260929 14:51:34.374831 18230 raft_consensus.cc:2454] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb8be85d17c8453d8fc369a0b6ab2809 in term 1.
I20260929 14:51:34.375236 18297 leader_election.cc:302] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06b0fd0792c471cb96b658ff77db280, cb8be85d17c8453d8fc369a0b6ab2809; no voters:
I20260929 14:51:34.375466 18431 raft_consensus.cc:2781] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:34.375754 18431 raft_consensus.cc:686] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 1 LEADER]: Becoming Leader. State: Replica: cb8be85d17c8453d8fc369a0b6ab2809, State: Running, Role: LEADER
I20260929 14:51:34.376269 18431 consensus_queue.cc:237] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } }
I20260929 14:51:34.379980 17957 catalog_manager.cc:5495] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb8be85d17c8453d8fc369a0b6ab2809 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8be85d17c8453d8fc369a0b6ab2809" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 } health_report { overall_health: UNKNOWN } } }
I20260929 14:51:34.463315 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.459214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.3:43915)
0929 14:51:34.459395 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:34.459410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:34.459424 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:51:34.459522 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:34.459524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:34.459587 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:34.459741 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:34.460035 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.460643 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.461616 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:34.461854 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:34.461969 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:34.462321 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:34.462849 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:34.462990 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:34.463032 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 14:51:34.463090 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:51:34.470413 18230 raft_consensus.cc:1261] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 1 FOLLOWER]: Refusing update from remote peer cb8be85d17c8453d8fc369a0b6ab2809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:34.470491 18099 raft_consensus.cc:1261] T aef99e0f87e44babb6f48c17d0a6251c P b06b0fd0792c471cb96b658ff77db280 [term 1 FOLLOWER]: Refusing update from remote peer cb8be85d17c8453d8fc369a0b6ab2809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:34.471352 18434 consensus_queue.cc:1035] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06b0fd0792c471cb96b658ff77db280" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:34.471593 18431 consensus_queue.cc:1035] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c750b2a518d34c6bb2d301627235112d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:34.489338 17957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:34.489691 17957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:34.492684 17957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a8799cbf0294142a243cf49906e0467: Sending DeleteTablet for 3 replicas of tablet aef99e0f87e44babb6f48c17d0a6251c
I20260929 14:51:34.493666 18210 tablet_service.cc:1514] Processing DeleteTablet for tablet aef99e0f87e44babb6f48c17d0a6251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:34 UTC) from {username='slave'} at 127.0.0.1:49468
I20260929 14:51:34.493674 18079 tablet_service.cc:1514] Processing DeleteTablet for tablet aef99e0f87e44babb6f48c17d0a6251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:34 UTC) from {username='slave'} at 127.0.0.1:57358
I20260929 14:51:34.494374 18447 tablet_replica.cc:330] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: stopping tablet replica
I20260929 14:51:34.494416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18017
I20260929 14:51:34.494405 18341 tablet_service.cc:1514] Processing DeleteTablet for tablet aef99e0f87e44babb6f48c17d0a6251c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:34 UTC) from {username='slave'} at 127.0.0.1:37546
I20260929 14:51:34.496675 18449 tablet_replica.cc:330] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: stopping tablet replica
I20260929 14:51:34.497004 18449 raft_consensus.cc:2229] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:51:34.497385 18449 raft_consensus.cc:2258] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:34.497694 18447 raft_consensus.cc:2229] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:34.498082 18447 raft_consensus.cc:2258] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:34.499454 18449 ts_tablet_manager.cc:1905] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:34.499856 18447 ts_tablet_manager.cc:1905] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:34.503376 18449 ts_tablet_manager.cc:1918] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:34.503511 18449 log.cc:1196] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/aef99e0f87e44babb6f48c17d0a6251c
W20260929 14:51:34.503688 17944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34619 (error 108)
I20260929 14:51:34.503875 18449 ts_tablet_manager.cc:1939] T aef99e0f87e44babb6f48c17d0a6251c P cb8be85d17c8453d8fc369a0b6ab2809: Deleting consensus metadata
I20260929 14:51:34.504056 18447 ts_tablet_manager.cc:1918] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:34.504266 18447 log.cc:1196] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/aef99e0f87e44babb6f48c17d0a6251c
I20260929 14:51:34.504617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18148
I20260929 14:51:34.504789 18447 ts_tablet_manager.cc:1939] T aef99e0f87e44babb6f48c17d0a6251c P c750b2a518d34c6bb2d301627235112d: Deleting consensus metadata
I20260929 14:51:34.505482 17943 catalog_manager.cc:4841] TS cb8be85d17c8453d8fc369a0b6ab2809 (127.0.98.3:43915): tablet aef99e0f87e44babb6f48c17d0a6251c (table test-table [id=2be2f88ecaac48b9ac5bfede1b810379]) successfully deleted
I20260929 14:51:34.506924 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.506528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34619 (local address 127.0.0.1:57370)
0929 14:51:34.506726 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.506816 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260929 14:51:34.507148 17944 catalog_manager.cc:4584] TS b06b0fd0792c471cb96b658ff77db280 (127.0.98.1:34619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aef99e0f87e44babb6f48c17d0a6251c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34619: connect: Connection refused (error 111)
I20260929 14:51:34.512482 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:34.512167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45747 (local address 127.0.0.1:49480)
0929 14:51:34.512309 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:34.512372 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 14:51:34.512699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18279
I20260929 14:51:34.519975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 17926
2026-09-29T14:51:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1398 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:34 dist-test-slave-5hd1 krb5kdc[18453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:34 dist-test-slave-5hd1 krb5kdc[18453](info): set up 2 sockets
Sep 29 14:51:34 dist-test-slave-5hd1 krb5kdc[18453](info): commencing operation
krb5kdc: starting...
W20260929 14:51:35.604221 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:35 dist-test-slave-5hd1 krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:51:35.740617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:35.853188 18466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:35.853513 18466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:35.853588 18466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:35.857649 18466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:35.857750 18466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:35.857789 18466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:35.857849 18466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:35.857887 18466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:35.863462 18466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:35.864586 18466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:35.865607 18466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:35.874842 18474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:35.874886 18471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:35.874876 18466 server_base.cc:1034] running on GCE node
W20260929 14:51:35.874842 18472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:35.875872 18466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:35.877036 18466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:35.878216 18466 hybrid_clock.cc:648] HybridClock initialized: now 1790693495878167 us; error 68 us; skew 500 ppm
Sep 29 14:51:35 dist-test-slave-5hd1 krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:35.886626 18466 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:51:35.887620 18466 webserver.cc:460] Webserver started at http://127.0.98.62:34731/ using document root <none> and password file <none>
I20260929 14:51:35.888267 18466 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:35.888360 18466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:35.888585 18466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:35.890720 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "6fd40a807aa240489f3f11c9c4136ec5"
format_stamp: "Formatted at 2026-09-29 14:51:35 on dist-test-slave-5hd1"
server_key: "4116a3cd20c93109681ed4bbbab2ac3f"
server_key_iv: "34d0a1c63179eeb23a7c9ce464cb67b5"
server_key_version: "encryptionkey@0"
I20260929 14:51:35.891287 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fd40a807aa240489f3f11c9c4136ec5"
format_stamp: "Formatted at 2026-09-29 14:51:35 on dist-test-slave-5hd1"
server_key: "4116a3cd20c93109681ed4bbbab2ac3f"
server_key_iv: "34d0a1c63179eeb23a7c9ce464cb67b5"
server_key_version: "encryptionkey@0"
I20260929 14:51:35.895238 18466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:51:35.898020 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:35.899662 18466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:35.899839 18466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "6fd40a807aa240489f3f11c9c4136ec5"
format_stamp: "Formatted at 2026-09-29 14:51:35 on dist-test-slave-5hd1"
server_key: "4116a3cd20c93109681ed4bbbab2ac3f"
server_key_iv: "34d0a1c63179eeb23a7c9ce464cb67b5"
server_key_version: "encryptionkey@0"
I20260929 14:51:35.899971 18466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:35.926357 18466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:35.929721 18466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:35.929945 18466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:35.937884 18466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40135
I20260929 14:51:35.937945 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40135 every 8 connection(s)
I20260929 14:51:35.939021 18466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:35.942616 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:35.948802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18466
I20260929 14:51:35.948918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:35.949270 392 external_mini_cluster.cc:1428] Setting key 6b3c89e70ae31b234234fe9190988615
I20260929 14:51:35.949466 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: Bootstrap starting.
I20260929 14:51:35.952521 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:35.953423 18534 log.cc:824] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:35.955572 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: No bootstrap required, opened a new log
Sep 29 14:51:35 dist-test-slave-5hd1 krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:35.960404 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260929 14:51:35.960697 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:35.960762 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd40a807aa240489f3f11c9c4136ec5, State: Initialized, Role: FOLLOWER
I20260929 14:51:35.961299 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260929 14:51:35.961443 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:35.961539 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:35.961674 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:35.962766 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260929 14:51:35.963207 18534 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd40a807aa240489f3f11c9c4136ec5; no voters:
I20260929 14:51:35.963608 18534 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:35.963804 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:35.964160 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd40a807aa240489f3f11c9c4136ec5, State: Running, Role: LEADER
I20260929 14:51:35.964710 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260929 14:51:35.965201 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:35.966125 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:35.950744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.98.62:40135)
0929 14:51:35.951373 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:35.951393 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:35.951444 (+ 51us) server_negotiation.cc:407] Connection header received
0929 14:51:35.952378 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:35.952400 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:35.952871 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:35.953248 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:35.954048 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:35.955081 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:35.955998 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:35.956560 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:35.956748 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:35.959822 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:35.959828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:35.959844 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:35.959875 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:35.962007 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:35.962482 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:35.962489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:35.962495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:35.962624 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:35.963102 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:35.963107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:35.963109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:35.963807 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:35.964057 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:35.964687 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 14:51:35.965097 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":207,"threads_started":1}
I20260929 14:51:35.967155 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd40a807aa240489f3f11c9c4136ec5. Latest consensus state: current_term: 1 leader_uuid: "6fd40a807aa240489f3f11c9c4136ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } } }
I20260929 14:51:35.967690 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:35.967175 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd40a807aa240489f3f11c9c4136ec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd40a807aa240489f3f11c9c4136ec5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } } }
I20260929 14:51:35.969506 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5 [sys.catalog]: This master's current role is: LEADER
W20260929 14:51:35.970800 18553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:51:35.970884 18553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:51:35.970971 18554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:35.971799 18554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:35.978230 18554 catalog_manager.cc:1347] Generated new cluster ID: 7e3806406a3744289c8cd83c2fe3c61e
I20260929 14:51:35.978312 18554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:35.998757 18554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:36.000264 18554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:36.009074 18554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: Generated new TSK 0
I20260929 14:51:36.009909 18554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:51:36.052011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:36.181288 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:36.181644 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:36.181726 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:36.185891 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:36.185950 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:36.186009 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:36.191165 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:36.192113 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:36.193143 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:36.203107 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.203188 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.203217 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:36.203336 18560 server_base.cc:1034] running on GCE node
I20260929 14:51:36.203965 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:36.204720 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:36.205911 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790693496205868 us; error 66 us; skew 500 ppm
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:36.208789 18560 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:51:36.209781 18560 webserver.cc:460] Webserver started at http://127.0.98.1:45697/ using document root <none> and password file <none>
I20260929 14:51:36.210438 18560 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:36.210518 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:36.210719 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:36.212834 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6596b1ea73b4617a077d185ed65f194"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "70d44d01f1c366ac44f70b967bb541b5"
server_key_iv: "729c1e0eb0e165fe657dc744d4a646f6"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.213342 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6596b1ea73b4617a077d185ed65f194"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "70d44d01f1c366ac44f70b967bb541b5"
server_key_iv: "729c1e0eb0e165fe657dc744d4a646f6"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.217362 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:36.220078 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.221624 18560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:51:36.221752 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "f6596b1ea73b4617a077d185ed65f194"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "70d44d01f1c366ac44f70b967bb541b5"
server_key_iv: "729c1e0eb0e165fe657dc744d4a646f6"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.221907 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:36.233636 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:36.236920 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:36.237121 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:36.237756 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:36.238624 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:36.238677 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.238718 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:36.238760 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.249516 18560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46151
I20260929 14:51:36.249703 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46151 every 8 connection(s)
I20260929 14:51:36.250537 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:36.259801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18560
I20260929 14:51:36.259948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:36.260423 392 external_mini_cluster.cc:1428] Setting key 5afe672bdbe94c866edd21bc519f6b9f
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:36.266997 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.252984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38811 (local address 127.0.98.62:40135)
0929 14:51:36.253165 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.253169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.254098 (+ 929us) server_negotiation.cc:407] Connection header received
0929 14:51:36.255584 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.255587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.255643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.255731 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:36.257155 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.257723 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.258995 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.259272 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.259365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.262357 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:36.262363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:36.262369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:36.262401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:36.264217 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.264996 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.264999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.265001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.265084 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.265575 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.265578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.265580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.265802 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:36.265945 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.266730 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.266848 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 14:51:36.268541 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.253312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40135 (local address 127.0.98.1:38811)
0929 14:51:36.253890 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.253945 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.255348 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.255939 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.255946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.256405 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:36.256988 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.257003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.257862 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.257865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.258850 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.258859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.259444 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.259447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.259608 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.260433 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:36.260451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:36.262171 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:36.264357 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.264368 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.264391 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.264849 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.265188 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.265194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.265198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.265444 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.265989 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:36.266001 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:36.266434 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.266676 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.267142 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260929 14:51:36.269961 18689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40135
I20260929 14:51:36.270300 18689 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:36.271071 18689 heartbeater.cc:511] Master 127.0.98.62:40135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:51:36.273134 18494 ts_manager.cc:194] Registered new tserver with Master: f6596b1ea73b4617a077d185ed65f194 (127.0.98.1:46151)
I20260929 14:51:36.274564 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:51:36.296815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:36.415024 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:36.415300 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:36.415343 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:36.419366 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:36.419489 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:36.419598 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:36.425230 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:36.426414 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:36.427467 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:36.437849 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.437826 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.437873 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:36.438154 18694 server_base.cc:1034] running on GCE node
I20260929 14:51:36.438673 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:36.439355 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:36.442654 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790693496442605 us; error 66 us; skew 500 ppm
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:36.446132 18694 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:51:36.447213 18694 webserver.cc:460] Webserver started at http://127.0.98.2:45143/ using document root <none> and password file <none>
I20260929 14:51:36.447858 18694 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:36.447927 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:36.448153 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:36.450328 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "f446b9226e294174a5ac3f9e49f15a4d"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "487bf50379100a182f009f20accd28c7"
server_key_iv: "b44c3a9ead92ab70a71a7140cf46462b"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.450920 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f446b9226e294174a5ac3f9e49f15a4d"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "487bf50379100a182f009f20accd28c7"
server_key_iv: "b44c3a9ead92ab70a71a7140cf46462b"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.455209 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:51:36.458069 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.459506 18694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:36.459689 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "f446b9226e294174a5ac3f9e49f15a4d"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "487bf50379100a182f009f20accd28c7"
server_key_iv: "b44c3a9ead92ab70a71a7140cf46462b"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.459816 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:36.474442 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:36.477736 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:36.477950 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:36.478556 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:36.479399 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:36.479471 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.479537 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:36.479585 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.490057 18694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37545
I20260929 14:51:36.490132 18822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37545 every 8 connection(s)
I20260929 14:51:36.491030 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:36.495271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18694
I20260929 14:51:36.495414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:36.495767 392 external_mini_cluster.cc:1428] Setting key 6251df29533a2032052ab50a86e702ed
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:51:36.509142 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.493365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49827 (local address 127.0.98.62:40135)
0929 14:51:36.493552 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.493556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.494380 (+ 824us) server_negotiation.cc:407] Connection header received
0929 14:51:36.496384 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.496388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.496455 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.496549 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:36.498145 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.498677 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.500334 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.500587 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.500723 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.503966 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:36.503972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:36.503976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:36.504014 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:36.506274 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.506958 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.506963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.506967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.507065 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.507594 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.507599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.507602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.507922 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:36.508033 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.508837 (+ 804us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.508988 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:51:36.510128 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.493685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40135 (local address 127.0.98.2:49827)
0929 14:51:36.494192 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.494242 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.496044 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.496749 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.496757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.497186 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:36.497960 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.497988 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.498816 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.498821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.500114 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.500124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.500776 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.500780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.500974 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.501709 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:36.501728 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:36.503694 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:36.506448 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.506454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.506469 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.506824 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.507318 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.507321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.507322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.507475 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.508049 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:36.508056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:36.508476 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.508742 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.509094 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:51:36.511981 18823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40135
I20260929 14:51:36.512392 18823 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:36.513204 18823 heartbeater.cc:511] Master 127.0.98.62:40135 requested a full tablet report, sending...
I20260929 14:51:36.514707 18494 ts_manager.cc:194] Registered new tserver with Master: f446b9226e294174a5ac3f9e49f15a4d (127.0.98.2:37545)
I20260929 14:51:36.515383 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:51:36.532059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:36.678871 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:36.679231 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:36.679318 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:36.683730 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:36.683858 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:36.683964 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:36.690064 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:36.691517 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:36.692850 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:36.704342 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.704406 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:36.704367 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:36.704759 18828 server_base.cc:1034] running on GCE node
I20260929 14:51:36.705433 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:36.706254 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:36.707449 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790693496707403 us; error 66 us; skew 500 ppm
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:36.710853 18828 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:51:36.712033 18828 webserver.cc:460] Webserver started at http://127.0.98.3:35227/ using document root <none> and password file <none>
I20260929 14:51:36.712785 18828 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:36.712893 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:36.713162 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:36.715585 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "bc5663cdcdc34edeada8a5be1e6d5a95"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "100b0b2987395b656561b75127d17291"
server_key_iv: "d9d55016ebb860c088ab8ec720559b33"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.716297 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bc5663cdcdc34edeada8a5be1e6d5a95"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "100b0b2987395b656561b75127d17291"
server_key_iv: "d9d55016ebb860c088ab8ec720559b33"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.721304 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:51:36.724314 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.726033 18828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:51:36.726233 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "bc5663cdcdc34edeada8a5be1e6d5a95"
format_stamp: "Formatted at 2026-09-29 14:51:36 on dist-test-slave-5hd1"
server_key: "100b0b2987395b656561b75127d17291"
server_key_iv: "d9d55016ebb860c088ab8ec720559b33"
server_key_version: "encryptionkey@0"
I20260929 14:51:36.726372 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:36.743153 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:36.746415 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:36.746573 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:36.747119 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:36.747984 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:36.748032 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.748070 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:36.748133 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:36.758188 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44927
I20260929 14:51:36.758247 18956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44927 every 8 connection(s)
I20260929 14:51:36.759085 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:36.760430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 18828
I20260929 14:51:36.760536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:36.760941 392 external_mini_cluster.cc:1428] Setting key 3a212103ad13714f4f4b9d7b0dfb58bb
Sep 29 14:51:36 dist-test-slave-5hd1 krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:36.774221 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.761432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47075 (local address 127.0.98.62:40135)
0929 14:51:36.761633 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.761639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.762449 (+ 810us) server_negotiation.cc:407] Connection header received
0929 14:51:36.764034 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.764037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.764092 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.764220 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:36.765381 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.765999 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.767334 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.767602 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.767692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.770317 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:36.770321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:36.770323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:36.770347 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:36.771874 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.772453 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.772456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.772458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.772547 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.772926 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.772930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.772931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.773143 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:36.773246 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.773811 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.773983 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 14:51:36.775027 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.761809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40135 (local address 127.0.98.3:47075)
0929 14:51:36.762275 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.762325 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.763823 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.764371 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.764378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.764715 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:36.765248 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.765262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.766122 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.766125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.767224 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.767233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.767792 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.767795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.767968 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.768669 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:36.768684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:36.770127 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:36.771990 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.771995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.772005 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.772336 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.772658 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:36.772661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:36.772662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:36.772791 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:36.773272 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:36.773277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:36.773501 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.773728 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.774036 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":144,"threads_started":1}
I20260929 14:51:36.776434 18957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40135
I20260929 14:51:36.776736 18957 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:36.777259 18957 heartbeater.cc:511] Master 127.0.98.62:40135 requested a full tablet report, sending...
I20260929 14:51:36.778442 18494 ts_manager.cc:194] Registered new tserver with Master: bc5663cdcdc34edeada8a5be1e6d5a95 (127.0.98.3:44927)
I20260929 14:51:36.779103 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47075
I20260929 14:51:36.788084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:36.799088 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.790401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.98.62:40135)
0929 14:51:36.790562 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.790566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.790839 (+ 273us) server_negotiation.cc:407] Connection header received
0929 14:51:36.790965 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.790968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.791017 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.791097 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:36.792049 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.792687 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.793591 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.793793 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.793873 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.795600 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:36.795603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:36.795606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:36.795628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:36.797216 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.797773 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.797776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.797777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.797894 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:36.798240 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:36.798243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:36.798244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:36.798441 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:36.798573 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.798866 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.798949 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 14:51:36.802388 18494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:36.805751 18494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:36.821533 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.815426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37545 (local address 127.0.0.1:58280)
0929 14:51:36.816121 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.816193 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.816328 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.817141 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.817148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.817171 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:36.817295 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.817301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.818578 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.818583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.819896 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.819907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.820845 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.820849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.820972 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.821010 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.821152 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.821344 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":25,"thread_start_us":170,"threads_started":1}
I20260929 14:51:36.821587 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.815615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.98.2:37545)
0929 14:51:36.816640 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.816651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.816675 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:51:36.816762 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.816767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.816967 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.817438 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:36.817597 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.818459 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.820039 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.820691 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.820848 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.821003 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.821329 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.821423 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":139,"threads_started":1}
I20260929 14:51:36.822125 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.815601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:60064)
0929 14:51:36.816575 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.816588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.816677 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.817290 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.817293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.817309 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:36.817407 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.817414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.818945 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.818949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.820055 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.820063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.821187 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.821191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.821785 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.821817 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.821904 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.822002 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260929 14:51:36.822319 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.815810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.98.1:46151)
0929 14:51:36.816438 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.816448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.816768 (+ 320us) server_negotiation.cc:407] Connection header received
0929 14:51:36.816842 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.816847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.816994 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.817126 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:36.817562 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.818813 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.820201 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.821046 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.821195 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.821378 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.821990 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.822147 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":105,"threads_started":1}
I20260929 14:51:36.822885 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.816375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44927 (local address 127.0.0.1:59988)
0929 14:51:36.817114 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.817129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.817247 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.817991 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.817994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.818013 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:36.818091 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.818097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.819402 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.819406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.820675 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.820686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.822437 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.822440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.822544 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.822565 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.822656 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.822747 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":120,"threads_started":1}
I20260929 14:51:36.823350 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.816705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.0.98.3:44927)
0929 14:51:36.817434 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.817445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.817465 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:51:36.817577 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.817582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.817813 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.817992 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:36.818197 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.819199 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.821155 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.822090 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.822205 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.822363 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.823071 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.823179 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":114,"threads_started":1}
I20260929 14:51:36.825575 18757 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 (DEFAULT_TABLE table=test-table [id=3f46bee3082c46638c8bc4101be40616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:36.825573 18623 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 (DEFAULT_TABLE table=test-table [id=3f46bee3082c46638c8bc4101be40616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:36.826855 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d0d09f29a4329b553e9101fca87c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:36.826855 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d0d09f29a4329b553e9101fca87c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:36.826458 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 (DEFAULT_TABLE table=test-table [id=3f46bee3082c46638c8bc4101be40616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:36.827705 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d0d09f29a4329b553e9101fca87c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:36.835355 18975 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Bootstrap starting.
I20260929 14:51:36.835669 18976 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Bootstrap starting.
I20260929 14:51:36.835987 18974 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: Bootstrap starting.
I20260929 14:51:36.837677 18976 tablet_bootstrap.cc:654] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:36.838131 18974 tablet_bootstrap.cc:654] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:36.838832 18976 log.cc:824] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:36.838939 18974 log.cc:824] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:36.841014 18976 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: No bootstrap required, opened a new log
I20260929 14:51:36.841220 18976 ts_tablet_manager.cc:1397] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:51:36.841578 18974 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: No bootstrap required, opened a new log
I20260929 14:51:36.841715 18975 tablet_bootstrap.cc:654] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:36.841841 18974 ts_tablet_manager.cc:1397] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:51:36.842561 18975 log.cc:824] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:36.844763 18975 tablet_bootstrap.cc:492] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: No bootstrap required, opened a new log
I20260929 14:51:36.844977 18975 ts_tablet_manager.cc:1397] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 14:51:36.845422 18976 raft_consensus.cc:348] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.845733 18976 raft_consensus.cc:374] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:36.845798 18976 raft_consensus.cc:729] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5663cdcdc34edeada8a5be1e6d5a95, State: Initialized, Role: FOLLOWER
I20260929 14:51:36.846273 18976 consensus_queue.cc:260] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.846139 18974 raft_consensus.cc:348] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.846764 18974 raft_consensus.cc:374] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:36.846822 18974 raft_consensus.cc:729] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6596b1ea73b4617a077d185ed65f194, State: Initialized, Role: FOLLOWER
I20260929 14:51:36.847333 18976 ts_tablet_manager.cc:1428] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:51:36.847553 18974 consensus_queue.cc:260] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.847726 18957 heartbeater.cc:503] Master 127.0.98.62:40135 was elected leader, sending a full tablet report...
I20260929 14:51:36.848356 18689 heartbeater.cc:503] Master 127.0.98.62:40135 was elected leader, sending a full tablet report...
I20260929 14:51:36.848784 18975 raft_consensus.cc:348] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.849026 18975 raft_consensus.cc:374] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:36.849066 18975 raft_consensus.cc:729] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f446b9226e294174a5ac3f9e49f15a4d, State: Initialized, Role: FOLLOWER
I20260929 14:51:36.849457 18975 consensus_queue.cc:260] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.850196 18975 ts_tablet_manager.cc:1428] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:51:36.851219 18823 heartbeater.cc:503] Master 127.0.98.62:40135 was elected leader, sending a full tablet report...
I20260929 14:51:36.852382 18974 ts_tablet_manager.cc:1428] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260929 14:51:36.879818 18980 raft_consensus.cc:482] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:36.880132 18980 raft_consensus.cc:504] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.881721 18980 leader_election.cc:288] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f446b9226e294174a5ac3f9e49f15a4d (127.0.98.2:37545), f6596b1ea73b4617a077d185ed65f194 (127.0.98.1:46151)
I20260929 14:51:36.887357 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.882239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60785 (local address 127.0.98.2:37545)
0929 14:51:36.882452 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.882456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.882470 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:51:36.882558 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.882563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.882622 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.882705 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:36.883071 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.883941 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.885951 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.886609 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.886765 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.886833 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.887122 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.887204 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 14:51:36.887357 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.882257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39887 (local address 127.0.98.1:46151)
0929 14:51:36.882411 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.882416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.882576 (+ 160us) server_negotiation.cc:407] Connection header received
0929 14:51:36.882717 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.882719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.882769 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.882853 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:36.883141 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.883938 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.885951 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.886640 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.886747 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.886827 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.887122 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.887205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:51:36.887336 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.881972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37545 (local address 127.0.98.3:60785)
0929 14:51:36.882120 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.882139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.882246 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.882754 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.882756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.882770 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:36.882947 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.882954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.884061 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.884064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.885790 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.885798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.886773 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.886777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.886944 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.886992 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.887077 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.887155 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":294}
I20260929 14:51:36.887348 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.882100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.98.3:39887)
0929 14:51:36.882470 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.882483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:36.882592 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:36.882866 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:36.882869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:36.882884 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:36.883015 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.883025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.884061 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.884064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.885787 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:36.885798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.886773 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.886777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:36.886926 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.886992 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 14:51:36.887077 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 14:51:36.887155 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":253,"thread_start_us":141,"threads_started":1}
I20260929 14:51:36.888339 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d0d09f29a4329b553e9101fca87c6" candidate_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f446b9226e294174a5ac3f9e49f15a4d" is_pre_election: true
I20260929 14:51:36.888334 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d0d09f29a4329b553e9101fca87c6" candidate_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6596b1ea73b4617a077d185ed65f194" is_pre_election: true
I20260929 14:51:36.888762 18643 raft_consensus.cc:2454] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc5663cdcdc34edeada8a5be1e6d5a95 in term 0.
I20260929 14:51:36.888754 18777 raft_consensus.cc:2454] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc5663cdcdc34edeada8a5be1e6d5a95 in term 0.
I20260929 14:51:36.889377 18846 leader_election.cc:302] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bc5663cdcdc34edeada8a5be1e6d5a95, f446b9226e294174a5ac3f9e49f15a4d, f6596b1ea73b4617a077d185ed65f194; no voters:
I20260929 14:51:36.889701 18980 raft_consensus.cc:2781] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:36.889791 18980 raft_consensus.cc:482] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:36.889832 18980 raft_consensus.cc:3037] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:36.891095 18980 raft_consensus.cc:504] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.891494 18980 leader_election.cc:288] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [CANDIDATE]: Term 1 election: Requested vote from peers f446b9226e294174a5ac3f9e49f15a4d (127.0.98.2:37545), f6596b1ea73b4617a077d185ed65f194 (127.0.98.1:46151)
I20260929 14:51:36.891904 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d0d09f29a4329b553e9101fca87c6" candidate_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6596b1ea73b4617a077d185ed65f194"
I20260929 14:51:36.891960 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d0d09f29a4329b553e9101fca87c6" candidate_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f446b9226e294174a5ac3f9e49f15a4d"
I20260929 14:51:36.892033 18643 raft_consensus.cc:3037] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:36.892212 18777 raft_consensus.cc:3037] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:36.893325 18643 raft_consensus.cc:2454] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc5663cdcdc34edeada8a5be1e6d5a95 in term 1.
I20260929 14:51:36.893468 18777 raft_consensus.cc:2454] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc5663cdcdc34edeada8a5be1e6d5a95 in term 1.
I20260929 14:51:36.893671 18846 leader_election.cc:302] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc5663cdcdc34edeada8a5be1e6d5a95, f6596b1ea73b4617a077d185ed65f194; no voters:
I20260929 14:51:36.893923 18980 raft_consensus.cc:2781] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:36.894145 18980 raft_consensus.cc:686] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 1 LEADER]: Becoming Leader. State: Replica: bc5663cdcdc34edeada8a5be1e6d5a95, State: Running, Role: LEADER
I20260929 14:51:36.894590 18980 consensus_queue.cc:237] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } }
I20260929 14:51:36.898379 18494 catalog_manager.cc:5495] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc5663cdcdc34edeada8a5be1e6d5a95 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc5663cdcdc34edeada8a5be1e6d5a95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20260929 14:51:36.925139 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.921503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.0.98.3:44927)
0929 14:51:36.921632 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:36.921635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:36.921651 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:51:36.921742 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:36.921744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:36.921801 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:36.921900 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:36.922155 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.922663 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.923573 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:36.923778 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:36.923859 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:36.924162 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:36.924650 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:36.924804 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:36.924906 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 14:51:36.924959 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:51:36.932052 18777 raft_consensus.cc:1261] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 1 FOLLOWER]: Refusing update from remote peer bc5663cdcdc34edeada8a5be1e6d5a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:36.932058 18643 raft_consensus.cc:1261] T 5a6d0d09f29a4329b553e9101fca87c6 P f6596b1ea73b4617a077d185ed65f194 [term 1 FOLLOWER]: Refusing update from remote peer bc5663cdcdc34edeada8a5be1e6d5a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:36.933017 18984 consensus_queue.cc:1035] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6596b1ea73b4617a077d185ed65f194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:36.933274 18980 consensus_queue.cc:1035] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f446b9226e294174a5ac3f9e49f15a4d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:36.943128 18989 mvcc.cc:205] Tried to move back new op lower bound from 7334680563425361920 to 7334680563283828736. Current Snapshot: MvccSnapshot[applied={T|T < 7334680563425361920}]
I20260929 14:51:36.946189 18988 mvcc.cc:205] Tried to move back new op lower bound from 7334680563425361920 to 7334680563283828736. Current Snapshot: MvccSnapshot[applied={T|T < 7334680563425361920}]
I20260929 14:51:36.946621 18987 mvcc.cc:205] Tried to move back new op lower bound from 7334680563425361920 to 7334680563283828736. Current Snapshot: MvccSnapshot[applied={T|T < 7334680563425361920 or (T in {7334680563425361920})}]
I20260929 14:51:36.951545 18494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:36.951735 18494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:36.954607 18494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fd40a807aa240489f3f11c9c4136ec5: Sending DeleteTablet for 3 replicas of tablet 5a6d0d09f29a4329b553e9101fca87c6
I20260929 14:51:36.955363 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58280
I20260929 14:51:36.955994 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59988
I20260929 14:51:36.956221 18997 tablet_replica.cc:330] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: stopping tablet replica
I20260929 14:51:36.956604 18997 raft_consensus.cc:2229] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:36.956905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18560
I20260929 14:51:36.956858 18623 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d0d09f29a4329b553e9101fca87c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60064
I20260929 14:51:36.956947 18997 raft_consensus.cc:2258] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:36.959841 18997 ts_tablet_manager.cc:1905] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:36.962072 18997 ts_tablet_manager.cc:1918] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:36.962193 18997 log.cc:1196] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/5a6d0d09f29a4329b553e9101fca87c6
I20260929 14:51:36.962492 18998 tablet_replica.cc:330] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: stopping tablet replica
I20260929 14:51:36.962531 18997 ts_tablet_manager.cc:1939] T 5a6d0d09f29a4329b553e9101fca87c6 P f446b9226e294174a5ac3f9e49f15a4d: Deleting consensus metadata
I20260929 14:51:36.962909 18998 raft_consensus.cc:2229] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:51:36.963333 18998 raft_consensus.cc:2258] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:36.964067 18483 catalog_manager.cc:4841] TS f446b9226e294174a5ac3f9e49f15a4d (127.0.98.2:37545): tablet 5a6d0d09f29a4329b553e9101fca87c6 (table test-table [id=3f46bee3082c46638c8bc4101be40616]) successfully deleted
I20260929 14:51:36.965090 18998 ts_tablet_manager.cc:1905] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:51:36.967151 18484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46151 (error 108)
I20260929 14:51:36.967474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18694
I20260929 14:51:36.968823 18998 ts_tablet_manager.cc:1918] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:36.968918 18998 log.cc:1196] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/5a6d0d09f29a4329b553e9101fca87c6
I20260929 14:51:36.969272 18998 ts_tablet_manager.cc:1939] T 5a6d0d09f29a4329b553e9101fca87c6 P bc5663cdcdc34edeada8a5be1e6d5a95: Deleting consensus metadata
I20260929 14:51:36.969941 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:36.969611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46151 (local address 127.0.0.1:60078)
0929 14:51:36.969790 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:36.969858 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 14:51:36.970178 18484 catalog_manager.cc:4584] TS f6596b1ea73b4617a077d185ed65f194 (127.0.98.1:46151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a6d0d09f29a4329b553e9101fca87c6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46151: connect: Connection refused (error 111)
I20260929 14:51:36.970475 18483 catalog_manager.cc:4841] TS bc5663cdcdc34edeada8a5be1e6d5a95 (127.0.98.3:44927): tablet 5a6d0d09f29a4329b553e9101fca87c6 (table test-table [id=3f46bee3082c46638c8bc4101be40616]) successfully deleted
I20260929 14:51:36.976985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18828
I20260929 14:51:36.983673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 18466
2026-09-29T14:51:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2464 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T14:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:37Z Disabled control of system clock
I20260929 14:51:37.017096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44627
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:37.131708 19003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:37.131982 19003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:37.132030 19003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:37.135968 19003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:37.136039 19003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:37.136054 19003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:37.136071 19003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:37.136086 19003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:37.140802 19003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:37.141844 19003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:37.143167 19003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:37.152416 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.152436 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:37.152714 19003 server_base.cc:1034] running on GCE node
W20260929 14:51:37.152436 19011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:37.153434 19003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:37.154575 19003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:37.155779 19003 hybrid_clock.cc:648] HybridClock initialized: now 1790693497155762 us; error 41 us; skew 500 ppm
I20260929 14:51:37.157783 19003 webserver.cc:460] Webserver started at http://127.0.98.62:36091/ using document root <none> and password file <none>
I20260929 14:51:37.158375 19003 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:37.158479 19003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:37.158732 19003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:37.160815 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "3d91aedafb53491c953744fd9a1ea774"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "8cfcc0193f5c46e86893f123f4ceaaec"
server_key_iv: "006791b16e11e7ee34c67ea894709098"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.161371 19003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d91aedafb53491c953744fd9a1ea774"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "8cfcc0193f5c46e86893f123f4ceaaec"
server_key_iv: "006791b16e11e7ee34c67ea894709098"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.165284 19003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:37.167936 19017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.169310 19003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:37.169461 19003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "3d91aedafb53491c953744fd9a1ea774"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "8cfcc0193f5c46e86893f123f4ceaaec"
server_key_iv: "006791b16e11e7ee34c67ea894709098"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.169610 19003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:37.180079 19003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:37.180750 19003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:37.180946 19003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:37.188555 19003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44627
I20260929 14:51:37.188575 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44627 every 8 connection(s)
I20260929 14:51:37.189492 19003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:37.193269 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:37.193675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19003
I20260929 14:51:37.193807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:37.194188 392 external_mini_cluster.cc:1428] Setting key a6d6ea3315766cc242b9db09dee480c6
I20260929 14:51:37.200480 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: Bootstrap starting.
I20260929 14:51:37.203058 19070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:37.203933 19070 log.cc:824] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:37.204277 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.195977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.98.62:44627)
0929 14:51:37.196739 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.196755 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.196794 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:51:37.197639 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.197667 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.198080 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.198441 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:37.198878 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.199890 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.200899 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.201376 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.201542 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.202437 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:37.202442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:37.202454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:37.202852 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:37.202907 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:37.202925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:37.203029 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.203139 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.203416 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":190,"threads_started":1}
I20260929 14:51:37.206388 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: No bootstrap required, opened a new log
I20260929 14:51:37.210016 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } }
I20260929 14:51:37.210233 19070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:37.210266 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d91aedafb53491c953744fd9a1ea774, State: Initialized, Role: FOLLOWER
I20260929 14:51:37.210614 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } }
I20260929 14:51:37.210736 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:37.210788 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:37.210860 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:37.211840 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } }
I20260929 14:51:37.212226 19070 leader_election.cc:302] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d91aedafb53491c953744fd9a1ea774; no voters:
I20260929 14:51:37.212584 19070 leader_election.cc:288] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:37.212776 19075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:37.213135 19075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [term 1 LEADER]: Becoming Leader. State: Replica: 3d91aedafb53491c953744fd9a1ea774, State: Running, Role: LEADER
I20260929 14:51:37.213533 19075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } }
I20260929 14:51:37.213811 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:37.215615 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d91aedafb53491c953744fd9a1ea774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } } }
I20260929 14:51:37.215775 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:37.216132 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d91aedafb53491c953744fd9a1ea774. Latest consensus state: current_term: 1 leader_uuid: "3d91aedafb53491c953744fd9a1ea774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d91aedafb53491c953744fd9a1ea774" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44627 } } }
I20260929 14:51:37.216342 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774 [sys.catalog]: This master's current role is: LEADER
W20260929 14:51:37.219398 19089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:51:37.219488 19089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:51:37.219559 19090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:37.220369 19090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:37.226627 19090 catalog_manager.cc:1347] Generated new cluster ID: 6ed26b3efedd4315b7d658cd25f3580c
I20260929 14:51:37.226704 19090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:37.238054 19090 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:37.239346 19090 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:37.244987 19090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: Generated new TSK 0
I20260929 14:51:37.245723 19090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:37.251550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:37.381238 19094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:37.381631 19094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:37.381723 19094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:37.386107 19094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:37.386201 19094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:37.386315 19094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:37.391888 19094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:37.393405 19094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:37.395107 19094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:37.406253 19102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.406236 19100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.406221 19099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:37.406610 19094 server_base.cc:1034] running on GCE node
I20260929 14:51:37.407120 19094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:37.407845 19094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:37.409045 19094 hybrid_clock.cc:648] HybridClock initialized: now 1790693497409022 us; error 38 us; skew 500 ppm
I20260929 14:51:37.411059 19094 webserver.cc:460] Webserver started at http://127.0.98.1:43323/ using document root <none> and password file <none>
I20260929 14:51:37.411689 19094 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:37.411779 19094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:37.411998 19094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:37.414263 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a1df8f20ca14e51b91828ba00c2c9a2"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "b18f76acc1da24f68e4b22e174b50700"
server_key_iv: "603f69b64645934200ee6c2f086a4663"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.414875 19094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a1df8f20ca14e51b91828ba00c2c9a2"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "b18f76acc1da24f68e4b22e174b50700"
server_key_iv: "603f69b64645934200ee6c2f086a4663"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.419042 19094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:51:37.421892 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.423357 19094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:37.423512 19094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "8a1df8f20ca14e51b91828ba00c2c9a2"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "b18f76acc1da24f68e4b22e174b50700"
server_key_iv: "603f69b64645934200ee6c2f086a4663"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.423640 19094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:37.449321 19094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:37.450050 19094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:37.450266 19094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:37.450891 19094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:37.451898 19094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:37.451972 19094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.452051 19094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:37.452095 19094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.462927 19094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42385
I20260929 14:51:37.462997 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42385 every 8 connection(s)
I20260929 14:51:37.463941 19094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:37.469436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19094
I20260929 14:51:37.469597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:37.470129 392 external_mini_cluster.cc:1428] Setting key 9ba55c86ebf00edca46108cb5e9f2d2a
I20260929 14:51:37.472812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:37.476543 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.466380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51111 (local address 127.0.98.62:44627)
0929 14:51:37.466552 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.466556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.467929 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 14:51:37.469666 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.469671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.469759 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.469868 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:37.471212 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.472035 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.473898 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.474230 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.474367 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.475954 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:37.475958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:37.475961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:37.475977 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:37.476009 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:37.476019 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:37.476095 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.476336 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.476419 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 14:51:37.478062 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.466864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44627 (local address 127.0.98.1:51111)
0929 14:51:37.467746 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.467796 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.469348 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.470135 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.470147 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.470272 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:37.471019 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.471041 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.472225 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.472229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.473759 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.473774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.474451 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.474459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.474661 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.475538 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:37.475542 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:37.475608 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:37.475716 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:37.476123 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:37.476136 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:37.476145 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.476332 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.476705 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":159,"threads_started":1}
I20260929 14:51:37.479794 19222 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44627
I20260929 14:51:37.480145 19222 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:37.480841 19222 heartbeater.cc:511] Master 127.0.98.62:44627 requested a full tablet report, sending...
I20260929 14:51:37.482617 19034 ts_manager.cc:194] Registered new tserver with Master: 8a1df8f20ca14e51b91828ba00c2c9a2 (127.0.98.1:42385)
I20260929 14:51:37.483807 19034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51111
W20260929 14:51:37.606773 19225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:37.607096 19225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:37.607169 19225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:37.611392 19225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:37.611469 19225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:37.611560 19225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:37.616909 19225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:37.618311 19225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:37.619985 19225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:37.631332 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.631317 19231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.631342 19230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:37.631851 19225 server_base.cc:1034] running on GCE node
I20260929 14:51:37.632352 19225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:37.633092 19225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:37.634294 19225 hybrid_clock.cc:648] HybridClock initialized: now 1790693497634249 us; error 62 us; skew 500 ppm
I20260929 14:51:37.636580 19225 webserver.cc:460] Webserver started at http://127.0.98.2:46323/ using document root <none> and password file <none>
I20260929 14:51:37.637260 19225 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:37.637326 19225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:37.637568 19225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:37.639882 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "df9f5253867141bd9aab10d626962fbe"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "81f3154ed8f3a44a38f4ff0501d40521"
server_key_iv: "b8ddf96b1d1b1f8efb85086cc86a9f4f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.640530 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "df9f5253867141bd9aab10d626962fbe"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "81f3154ed8f3a44a38f4ff0501d40521"
server_key_iv: "b8ddf96b1d1b1f8efb85086cc86a9f4f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.644773 19225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:37.647662 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.649246 19225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:37.649394 19225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "df9f5253867141bd9aab10d626962fbe"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "81f3154ed8f3a44a38f4ff0501d40521"
server_key_iv: "b8ddf96b1d1b1f8efb85086cc86a9f4f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.649525 19225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:37.664865 19225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:37.665753 19225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:37.665987 19225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:37.666710 19225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:37.667927 19225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:37.668002 19225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.668081 19225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:37.668136 19225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.679816 19225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34685
I20260929 14:51:37.679904 19352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34685 every 8 connection(s)
I20260929 14:51:37.681104 19225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:37.681744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19225
I20260929 14:51:37.681883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:37.682262 392 external_mini_cluster.cc:1428] Setting key abd93f64f2d98e6012ded52f2bfe2f0b
I20260929 14:51:37.684962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--builtin_ntp_servers=127.0.98.20:44293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:37.694178 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.683671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35793 (local address 127.0.98.62:44627)
0929 14:51:37.683815 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.683819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.685691 (+ 1872us) server_negotiation.cc:407] Connection header received
0929 14:51:37.687556 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.687560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.687622 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.687708 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:37.688981 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.689686 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.691666 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.691903 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.692001 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.693609 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:37.693613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:37.693615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:37.693632 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:37.693653 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:37.693661 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:37.693751 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.693958 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.694026 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:51:37.695508 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.684460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44627 (local address 127.0.98.2:35793)
0929 14:51:37.685487 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.685560 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.687316 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.687897 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.687909 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.688027 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:37.688804 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.688824 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.689847 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.689853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.691509 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.691523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.692098 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.692104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.692423 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.693221 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:37.693225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:37.693289 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:37.693389 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:37.693774 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:37.693786 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:37.693794 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.693938 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.694293 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":157,"threads_started":1}
I20260929 14:51:37.696959 19353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44627
I20260929 14:51:37.697307 19353 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:37.698011 19353 heartbeater.cc:511] Master 127.0.98.62:44627 requested a full tablet report, sending...
I20260929 14:51:37.699234 19034 ts_manager.cc:194] Registered new tserver with Master: df9f5253867141bd9aab10d626962fbe (127.0.98.2:34685)
I20260929 14:51:37.699911 19034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35793
W20260929 14:51:37.807469 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:37.807782 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:37.807859 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:37.811950 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:37.812026 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:37.812112 19356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:37.817057 19356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:37.818173 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:37.819556 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:37.830426 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.830428 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:37.830428 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:37.830897 19356 server_base.cc:1034] running on GCE node
I20260929 14:51:37.831480 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:37.832438 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:37.833667 19356 hybrid_clock.cc:648] HybridClock initialized: now 1790693497833629 us; error 53 us; skew 500 ppm
I20260929 14:51:37.835887 19356 webserver.cc:460] Webserver started at http://127.0.98.3:36337/ using document root <none> and password file <none>
I20260929 14:51:37.836683 19356 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:37.836790 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:37.837051 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:37.839565 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "364ba9907e034ba38314b96bbc83e763"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "04c41b8c3b4dd256ff5a6e9f536d774e"
server_key_iv: "db859aa532b5041bc78c07303dc5b97f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.840421 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "364ba9907e034ba38314b96bbc83e763"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "04c41b8c3b4dd256ff5a6e9f536d774e"
server_key_iv: "db859aa532b5041bc78c07303dc5b97f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.844738 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:37.847591 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.849120 19356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:37.849273 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "364ba9907e034ba38314b96bbc83e763"
format_stamp: "Formatted at 2026-09-29 14:51:37 on dist-test-slave-5hd1"
server_key: "04c41b8c3b4dd256ff5a6e9f536d774e"
server_key_iv: "db859aa532b5041bc78c07303dc5b97f"
server_key_version: "encryptionkey@0"
I20260929 14:51:37.849401 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:37.865082 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:37.865926 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:37.866140 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:37.866828 19356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:37.867944 19356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:37.868016 19356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.868090 19356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:37.868142 19356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:37.879637 19356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36749
I20260929 14:51:37.879658 19483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36749 every 8 connection(s)
I20260929 14:51:37.880839 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:37.883087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19356
I20260929 14:51:37.883215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:37.883579 392 external_mini_cluster.cc:1428] Setting key 2eee31a61167f87cd57044b579475d64
I20260929 14:51:37.891368 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.883093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53979 (local address 127.0.98.62:44627)
0929 14:51:37.883249 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.883253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.884653 (+ 1400us) server_negotiation.cc:407] Connection header received
0929 14:51:37.886256 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.886258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.886313 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.886386 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:37.887326 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.887921 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.889325 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.889544 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.889626 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.890870 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:37.890873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:37.890875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:37.890887 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:37.890905 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:37.890913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:37.890973 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.891174 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.891228 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:51:37.892350 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.883668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44627 (local address 127.0.98.3:53979)
0929 14:51:37.884477 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.884531 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.886019 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.886578 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.886585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.886665 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:37.887191 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.887204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.888109 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.888113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.889197 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.889208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.889689 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.889692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.889846 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.890610 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:37.890612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:37.890655 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:37.890722 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:37.890996 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:37.891028 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:37.891033 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.891153 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.891383 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":168,"threads_started":1}
I20260929 14:51:37.893596 19484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44627
I20260929 14:51:37.893819 19484 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:37.894331 19484 heartbeater.cc:511] Master 127.0.98.62:44627 requested a full tablet report, sending...
I20260929 14:51:37.895421 19034 ts_manager.cc:194] Registered new tserver with Master: 364ba9907e034ba38314b96bbc83e763 (127.0.98.3:36749)
I20260929 14:51:37.896099 19034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53979
I20260929 14:51:37.898209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:37.904966 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.900746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.98.62:44627)
0929 14:51:37.900914 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.900919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.901190 (+ 271us) server_negotiation.cc:407] Connection header received
0929 14:51:37.901312 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.901315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.901360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.901426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:37.901716 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.902246 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.903267 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.903479 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.903557 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.904471 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:37.904474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:37.904476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:37.904488 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:37.904508 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:37.904517 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:37.904618 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.904740 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.904775 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:51:37.907996 19034 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:37.911047 19034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:37.927950 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.920821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36749 (local address 127.0.0.1:54558)
0929 14:51:37.922268 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.922286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.922397 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.923004 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.923007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.923025 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:37.923119 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.923125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924916 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.924923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.926063 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.926073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.927284 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.927289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.927421 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.927473 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.927582 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.927743 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":81,"thread_start_us":102,"threads_started":1}
I20260929 14:51:37.928047 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.920915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.0.98.3:36749)
0929 14:51:37.921345 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.921359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.922501 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 14:51:37.922607 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.922618 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.922848 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.923027 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:37.923224 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924750 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.926203 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.927147 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.927292 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.927623 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.927686 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.927830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":206,"threads_started":1}
I20260929 14:51:37.928706 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.922141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.98.1:42385)
0929 14:51:37.922905 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.922915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.922938 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:51:37.923030 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.923035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.923209 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.923371 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:37.923626 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924799 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.926412 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.927189 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.927733 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.928352 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.928409 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.928527 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":117,"threads_started":1}
I20260929 14:51:37.928759 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.920808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34685 (local address 127.0.0.1:36722)
0929 14:51:37.921674 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.921754 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.921894 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.922773 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.922788 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.922813 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:37.922991 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.923000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924329 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.924333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.926548 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.926557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.928259 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.928264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.928394 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.928423 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.928504 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.928596 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":35,"thread_start_us":111,"threads_started":1}
I20260929 14:51:37.928910 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.921636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42385 (local address 127.0.0.1:54246)
0929 14:51:37.922583 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:37.922596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:37.922667 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:37.923367 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:37.923370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:37.923390 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:37.923490 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.923497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924948 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.924951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.926252 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:37.926263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.927352 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.927356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:37.927443 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.927473 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:51:37.927572 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 14:51:37.928771 (+ 1199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":1115,"thread_start_us":150,"threads_started":1}
I20260929 14:51:37.929328 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:37.921789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.0.98.2:34685)
0929 14:51:37.922244 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:37.922260 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:37.922300 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:51:37.922392 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:37.922398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:37.922597 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:37.922786 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:37.923132 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.924131 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.926709 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:37.927776 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:37.927914 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:37.928123 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:37.929030 (+ 907us) server_negotiation.cc:299] Negotiation successful
0929 14:51:37.929157 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260929 14:51:37.931870 19287 tablet_service.cc:1467] Processing CreateTablet for tablet d91736539be746ff8dd76c161298c113 (DEFAULT_TABLE table=test-table [id=c95483909b904686aae4221620d05e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:37.932265 19156 tablet_service.cc:1467] Processing CreateTablet for tablet d91736539be746ff8dd76c161298c113 (DEFAULT_TABLE table=test-table [id=c95483909b904686aae4221620d05e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:37.932265 19418 tablet_service.cc:1467] Processing CreateTablet for tablet d91736539be746ff8dd76c161298c113 (DEFAULT_TABLE table=test-table [id=c95483909b904686aae4221620d05e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:37.933156 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91736539be746ff8dd76c161298c113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:37.933420 19418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91736539be746ff8dd76c161298c113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:37.933420 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91736539be746ff8dd76c161298c113. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:37.940366 19501 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Bootstrap starting.
I20260929 14:51:37.942006 19502 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Bootstrap starting.
I20260929 14:51:37.942818 19503 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Bootstrap starting.
I20260929 14:51:37.943101 19501 tablet_bootstrap.cc:654] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:37.944089 19501 log.cc:824] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:37.944294 19502 tablet_bootstrap.cc:654] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:37.945127 19502 log.cc:824] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:37.945250 19503 tablet_bootstrap.cc:654] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:37.946185 19503 log.cc:824] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:37.946404 19501 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: No bootstrap required, opened a new log
I20260929 14:51:37.946626 19501 ts_tablet_manager.cc:1397] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260929 14:51:37.947309 19502 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: No bootstrap required, opened a new log
I20260929 14:51:37.947499 19502 ts_tablet_manager.cc:1397] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260929 14:51:37.948390 19503 tablet_bootstrap.cc:492] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: No bootstrap required, opened a new log
I20260929 14:51:37.948596 19503 ts_tablet_manager.cc:1397] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:37.951500 19501 raft_consensus.cc:348] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.951808 19501 raft_consensus.cc:374] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:37.951864 19501 raft_consensus.cc:729] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9f5253867141bd9aab10d626962fbe, State: Initialized, Role: FOLLOWER
I20260929 14:51:37.951999 19502 raft_consensus.cc:348] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.952250 19502 raft_consensus.cc:374] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:37.952301 19502 raft_consensus.cc:729] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364ba9907e034ba38314b96bbc83e763, State: Initialized, Role: FOLLOWER
I20260929 14:51:37.952440 19501 consensus_queue.cc:260] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.952322 19503 raft_consensus.cc:348] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.952639 19503 raft_consensus.cc:374] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:37.952695 19503 raft_consensus.cc:729] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1df8f20ca14e51b91828ba00c2c9a2, State: Initialized, Role: FOLLOWER
I20260929 14:51:37.952745 19502 consensus_queue.cc:260] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.953231 19503 consensus_queue.cc:260] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:37.953644 19501 ts_tablet_manager.cc:1428] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 14:51:37.953672 19502 ts_tablet_manager.cc:1428] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 14:51:37.953680 19484 heartbeater.cc:503] Master 127.0.98.62:44627 was elected leader, sending a full tablet report...
I20260929 14:51:37.954072 19353 heartbeater.cc:503] Master 127.0.98.62:44627 was elected leader, sending a full tablet report...
I20260929 14:51:37.955255 19503 ts_tablet_manager.cc:1428] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:51:37.955628 19222 heartbeater.cc:503] Master 127.0.98.62:44627 was elected leader, sending a full tablet report...
W20260929 14:51:37.965370 19223 tablet.cc:2369] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:38.059788 19509 raft_consensus.cc:482] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:38.060086 19509 raft_consensus.cc:504] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:38.061417 19509 leader_election.cc:288] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 364ba9907e034ba38314b96bbc83e763 (127.0.98.3:36749), df9f5253867141bd9aab10d626962fbe (127.0.98.2:34685)
I20260929 14:51:38.066876 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.061910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59249 (local address 127.0.98.3:36749)
0929 14:51:38.062180 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:38.062187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:38.062553 (+ 366us) server_negotiation.cc:407] Connection header received
0929 14:51:38.062733 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:38.062737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:38.062792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:38.062881 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:38.063244 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.063815 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.065637 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.066227 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.066366 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:38.066443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:38.066649 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 14:51:38.066733 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 14:51:38.066912 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.061887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36749 (local address 127.0.98.1:59249)
0929 14:51:38.062401 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:38.062415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:38.062579 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:38.062909 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:38.062911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:38.062928 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:38.063121 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:38.063128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.063947 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.063950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:38.065431 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:38.065441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.066392 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.066395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:38.066478 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:38.066498 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:51:38.066606 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:51:38.066701 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":99,"thread_start_us":199,"threads_started":1}
I20260929 14:51:38.066885 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.061944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44787 (local address 127.0.98.2:34685)
0929 14:51:38.062111 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:38.062119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:38.062319 (+ 200us) server_negotiation.cc:407] Connection header received
0929 14:51:38.062529 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:38.062533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:38.062596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:38.062685 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:38.063136 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.063705 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.065634 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.066085 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.066178 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:38.066247 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:38.066570 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 14:51:38.066674 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:51:38.066941 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.061844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34685 (local address 127.0.98.1:44787)
0929 14:51:38.062183 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:38.062202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:38.062386 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:38.062723 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:38.062726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:38.062747 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:38.063000 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:38.063009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.063858 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.063864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:38.065430 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:38.065441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.066250 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.066258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:38.066414 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:38.066442 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:51:38.066545 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 14:51:38.066693 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":95,"thread_start_us":107,"threads_started":1}
I20260929 14:51:38.067965 19307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91736539be746ff8dd76c161298c113" candidate_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9f5253867141bd9aab10d626962fbe" is_pre_election: true
I20260929 14:51:38.067986 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91736539be746ff8dd76c161298c113" candidate_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364ba9907e034ba38314b96bbc83e763" is_pre_election: true
I20260929 14:51:38.068357 19438 raft_consensus.cc:2454] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1df8f20ca14e51b91828ba00c2c9a2 in term 0.
I20260929 14:51:38.068357 19307 raft_consensus.cc:2454] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a1df8f20ca14e51b91828ba00c2c9a2 in term 0.
I20260929 14:51:38.068991 19109 leader_election.cc:302] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 364ba9907e034ba38314b96bbc83e763, 8a1df8f20ca14e51b91828ba00c2c9a2, df9f5253867141bd9aab10d626962fbe; no voters:
I20260929 14:51:38.069437 19509 raft_consensus.cc:2781] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:38.069569 19509 raft_consensus.cc:482] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:38.069636 19509 raft_consensus.cc:3037] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:38.070832 19509 raft_consensus.cc:504] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:38.071301 19509 leader_election.cc:288] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [CANDIDATE]: Term 1 election: Requested vote from peers 364ba9907e034ba38314b96bbc83e763 (127.0.98.3:36749), df9f5253867141bd9aab10d626962fbe (127.0.98.2:34685)
I20260929 14:51:38.071811 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91736539be746ff8dd76c161298c113" candidate_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "364ba9907e034ba38314b96bbc83e763"
I20260929 14:51:38.071962 19438 raft_consensus.cc:3037] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:38.071905 19307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91736539be746ff8dd76c161298c113" candidate_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df9f5253867141bd9aab10d626962fbe"
I20260929 14:51:38.072154 19307 raft_consensus.cc:3037] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:38.073342 19438 raft_consensus.cc:2454] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1df8f20ca14e51b91828ba00c2c9a2 in term 1.
I20260929 14:51:38.073773 19307 raft_consensus.cc:2454] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a1df8f20ca14e51b91828ba00c2c9a2 in term 1.
I20260929 14:51:38.073789 19109 leader_election.cc:302] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 364ba9907e034ba38314b96bbc83e763, 8a1df8f20ca14e51b91828ba00c2c9a2; no voters:
I20260929 14:51:38.073994 19509 raft_consensus.cc:2781] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:38.074363 19509 raft_consensus.cc:686] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 8a1df8f20ca14e51b91828ba00c2c9a2, State: Running, Role: LEADER
I20260929 14:51:38.074780 19509 consensus_queue.cc:237] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } }
I20260929 14:51:38.078727 19033 catalog_manager.cc:5495] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a1df8f20ca14e51b91828ba00c2c9a2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a1df8f20ca14e51b91828ba00c2c9a2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42385 } health_report { overall_health: HEALTHY } } }
I20260929 14:51:38.122771 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.118815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.98.1:42385)
0929 14:51:38.119021 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:38.119026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:38.119041 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:51:38.119126 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:38.119128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:38.119196 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:38.119341 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:38.119601 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.120145 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.121093 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:38.121331 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:38.121445 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:38.121809 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:38.122301 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:38.122473 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:38.122567 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 14:51:38.122608 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:51:38.130018 19438 raft_consensus.cc:1261] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 1 FOLLOWER]: Refusing update from remote peer 8a1df8f20ca14e51b91828ba00c2c9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:38.130012 19307 raft_consensus.cc:1261] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 1 FOLLOWER]: Refusing update from remote peer 8a1df8f20ca14e51b91828ba00c2c9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:38.130904 19509 consensus_queue.cc:1035] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9f5253867141bd9aab10d626962fbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:38.131181 19512 consensus_queue.cc:1035] T d91736539be746ff8dd76c161298c113 P 8a1df8f20ca14e51b91828ba00c2c9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "364ba9907e034ba38314b96bbc83e763" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:51:38.131909 19485 tablet.cc:2369] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:38.148756 19033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:38.148959 19033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:38.151721 19033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d91aedafb53491c953744fd9a1ea774: Sending DeleteTablet for 3 replicas of tablet d91736539be746ff8dd76c161298c113
I20260929 14:51:38.152694 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet d91736539be746ff8dd76c161298c113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:38 UTC) from {username='slave'} at 127.0.0.1:36722
I20260929 14:51:38.152941 19156 tablet_service.cc:1514] Processing DeleteTablet for tablet d91736539be746ff8dd76c161298c113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:38 UTC) from {username='slave'} at 127.0.0.1:54246
I20260929 14:51:38.152942 19418 tablet_service.cc:1514] Processing DeleteTablet for tablet d91736539be746ff8dd76c161298c113 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:38 UTC) from {username='slave'} at 127.0.0.1:54558
I20260929 14:51:38.153240 19525 tablet_replica.cc:330] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: stopping tablet replica
I20260929 14:51:38.153570 19525 raft_consensus.cc:2229] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:38.153970 19525 raft_consensus.cc:2258] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:38.153955 19526 tablet_replica.cc:330] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: stopping tablet replica
I20260929 14:51:38.154654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19094
I20260929 14:51:38.157136 19526 raft_consensus.cc:2229] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:38.157269 19525 ts_tablet_manager.cc:1905] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:38.157475 19526 raft_consensus.cc:2258] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:38.159227 19526 ts_tablet_manager.cc:1905] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:38.159829 19525 ts_tablet_manager.cc:1918] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:38.159952 19525 log.cc:1196] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/d91736539be746ff8dd76c161298c113
I20260929 14:51:38.160324 19525 ts_tablet_manager.cc:1939] T d91736539be746ff8dd76c161298c113 P df9f5253867141bd9aab10d626962fbe: Deleting consensus metadata
I20260929 14:51:38.161551 19021 catalog_manager.cc:4841] TS df9f5253867141bd9aab10d626962fbe (127.0.98.2:34685): tablet d91736539be746ff8dd76c161298c113 (table test-table [id=c95483909b904686aae4221620d05e76]) successfully deleted
I20260929 14:51:38.163264 19526 ts_tablet_manager.cc:1918] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:38.163406 19526 log.cc:1196] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/d91736539be746ff8dd76c161298c113
I20260929 14:51:38.163734 19526 ts_tablet_manager.cc:1939] T d91736539be746ff8dd76c161298c113 P 364ba9907e034ba38314b96bbc83e763: Deleting consensus metadata
W20260929 14:51:38.164250 19020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42385 (error 108)
I20260929 14:51:38.164572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19225
I20260929 14:51:38.165529 19018 catalog_manager.cc:4841] TS 364ba9907e034ba38314b96bbc83e763 (127.0.98.3:36749): tablet d91736539be746ff8dd76c161298c113 (table test-table [id=c95483909b904686aae4221620d05e76]) successfully deleted
I20260929 14:51:38.167371 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:38.166999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42385 (local address 127.0.0.1:54260)
0929 14:51:38.167187 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:38.167269 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 14:51:38.167577 19020 catalog_manager.cc:4584] TS 8a1df8f20ca14e51b91828ba00c2c9a2 (127.0.98.1:42385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d91736539be746ff8dd76c161298c113: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42385: connect: Connection refused (error 111)
I20260929 14:51:38.171545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19356
I20260929 14:51:38.178830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19003
2026-09-29T14:51:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:38 dist-test-slave-5hd1 krb5kdc[19531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:38 dist-test-slave-5hd1 krb5kdc[19531](info): set up 2 sockets
Sep 29 14:51:38 dist-test-slave-5hd1 krb5kdc[19531](info): commencing operation
krb5kdc: starting...
W20260929 14:51:39.300717 392 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:39 dist-test-slave-5hd1 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:51:39.439633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:39.566815 19544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:39.567157 19544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:39.567245 19544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:39.571354 19544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:39.571461 19544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:39.571501 19544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:39.571538 19544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:39.571576 19544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:39.577080 19544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:39.578363 19544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:39.579582 19544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:39.589426 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:39.589512 19550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:39.589435 19549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:39.589720 19544 server_base.cc:1034] running on GCE node
I20260929 14:51:39.590458 19544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:39.591727 19544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:39.592926 19544 hybrid_clock.cc:648] HybridClock initialized: now 1790693499592898 us; error 46 us; skew 500 ppm
Sep 29 14:51:39 dist-test-slave-5hd1 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:39.595911 19544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:51:39.596920 19544 webserver.cc:460] Webserver started at http://127.0.98.62:37013/ using document root <none> and password file <none>
I20260929 14:51:39.597574 19544 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:39.597666 19544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:39.597905 19544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:39.599994 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "f897dbc9531941f2aa43aabd01ee2976"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "880ecb3263697f3e230308049775a5cc"
server_key_iv: "bb2cfc987633aae5aeaad2e45b9a8ba2"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.600615 19544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "f897dbc9531941f2aa43aabd01ee2976"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "880ecb3263697f3e230308049775a5cc"
server_key_iv: "bb2cfc987633aae5aeaad2e45b9a8ba2"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.604650 19544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:39.607165 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:39.608640 19544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:39.608783 19544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "f897dbc9531941f2aa43aabd01ee2976"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "880ecb3263697f3e230308049775a5cc"
server_key_iv: "bb2cfc987633aae5aeaad2e45b9a8ba2"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.608904 19544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:39.619339 19544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:39.622260 19544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:39.622457 19544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:39.630293 19544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45665
I20260929 14:51:39.630460 19611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45665 every 8 connection(s)
I20260929 14:51:39.631312 19544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:39.634760 19612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:39.636559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19544
I20260929 14:51:39.636682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:39.637003 392 external_mini_cluster.cc:1428] Setting key a224e118494355140929222ebd5f8fe6
I20260929 14:51:39.642812 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: Bootstrap starting.
I20260929 14:51:39.645953 19612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:39.646817 19612 log.cc:824] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:51:39 dist-test-slave-5hd1 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:39.649801 19612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: No bootstrap required, opened a new log
I20260929 14:51:39.654843 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:39.638810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.0.98.62:45665)
0929 14:51:39.639337 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:39.639359 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:39.639415 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:51:39.640385 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:39.640413 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:39.640898 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:39.641328 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:39.642044 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.643563 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.644506 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.645217 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.645399 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:39.648626 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:39.648633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:39.648651 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:39.648685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:39.650984 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:39.651470 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:39.651477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:39.651484 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:39.651644 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:39.652001 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:39.652004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:39.652006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:39.652705 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:39.652973 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:39.653260 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 14:51:39.653631 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":138,"threads_started":1}
I20260929 14:51:39.655159 19612 raft_consensus.cc:348] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } }
I20260929 14:51:39.655485 19612 raft_consensus.cc:374] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:39.655546 19612 raft_consensus.cc:729] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f897dbc9531941f2aa43aabd01ee2976, State: Initialized, Role: FOLLOWER
I20260929 14:51:39.656081 19612 consensus_queue.cc:260] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } }
I20260929 14:51:39.656335 19612 raft_consensus.cc:388] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:39.656409 19612 raft_consensus.cc:482] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:39.656519 19612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:39.657805 19612 raft_consensus.cc:504] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } }
I20260929 14:51:39.658221 19612 leader_election.cc:302] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f897dbc9531941f2aa43aabd01ee2976; no voters:
I20260929 14:51:39.658603 19612 leader_election.cc:288] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:39.658789 19617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:39.659068 19617 raft_consensus.cc:686] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [term 1 LEADER]: Becoming Leader. State: Replica: f897dbc9531941f2aa43aabd01ee2976, State: Running, Role: LEADER
I20260929 14:51:39.659467 19617 consensus_queue.cc:237] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } }
I20260929 14:51:39.660156 19612 sys_catalog.cc:564] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:39.661317 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f897dbc9531941f2aa43aabd01ee2976. Latest consensus state: current_term: 1 leader_uuid: "f897dbc9531941f2aa43aabd01ee2976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } } }
I20260929 14:51:39.661516 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:39.662006 19618 sys_catalog.cc:455] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f897dbc9531941f2aa43aabd01ee2976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f897dbc9531941f2aa43aabd01ee2976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45665 } } }
I20260929 14:51:39.662148 19618 sys_catalog.cc:458] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:39.662956 19626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:39.666134 19626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:39.671747 19626 catalog_manager.cc:1347] Generated new cluster ID: 43479d3b5bab43b99053f72ce063ab0d
I20260929 14:51:39.671814 19626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:39.687781 19626 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:39.689064 19626 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:39.696457 19626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: Generated new TSK 0
I20260929 14:51:39.697211 19626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:51:39.748061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:39.865871 19638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:39.866134 19638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:39.866175 19638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:39.870141 19638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:39.870199 19638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:39.870296 19638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:39.876035 19638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:39.877185 19638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:39.878203 19638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:39.888137 19646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:39.888099 19644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:39.888090 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:39.888440 19638 server_base.cc:1034] running on GCE node
I20260929 14:51:39.888983 19638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:39.889573 19638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:39.890775 19638 hybrid_clock.cc:648] HybridClock initialized: now 1790693499890757 us; error 37 us; skew 500 ppm
Sep 29 14:51:39 dist-test-slave-5hd1 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:39.893699 19638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:51:39.894578 19638 webserver.cc:460] Webserver started at http://127.0.98.1:39103/ using document root <none> and password file <none>
I20260929 14:51:39.895103 19638 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:39.895164 19638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:39.895318 19638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:39.897266 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc4983714eb34a7bbbae0754697c0e4c"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "a3f316830b77c49f5efc3176cf5f9ea9"
server_key_iv: "562d40e94ce8d538c09a91fb488312a5"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.897753 19638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc4983714eb34a7bbbae0754697c0e4c"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "a3f316830b77c49f5efc3176cf5f9ea9"
server_key_iv: "562d40e94ce8d538c09a91fb488312a5"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.901430 19638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:51:39.903950 19653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:39.905388 19638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:39.905496 19638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "cc4983714eb34a7bbbae0754697c0e4c"
format_stamp: "Formatted at 2026-09-29 14:51:39 on dist-test-slave-5hd1"
server_key: "a3f316830b77c49f5efc3176cf5f9ea9"
server_key_iv: "562d40e94ce8d538c09a91fb488312a5"
server_key_version: "encryptionkey@0"
I20260929 14:51:39.905586 19638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:39.922099 19638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:39.925354 19638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:39.925501 19638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:39.926014 19638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:39.926836 19638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:39.926908 19638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:39.926985 19638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:39.927034 19638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:39.937284 19638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33453
I20260929 14:51:39.937304 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33453 every 8 connection(s)
I20260929 14:51:39.938148 19638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:39.944674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19638
I20260929 14:51:39.944777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:39.945088 392 external_mini_cluster.cc:1428] Setting key 89d93ca9215deeb574d61b5ce575b483
Sep 29 14:51:39 dist-test-slave-5hd1 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:39.954113 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:39.940359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43193 (local address 127.0.98.62:45665)
0929 14:51:39.940536 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:39.940541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:39.941442 (+ 901us) server_negotiation.cc:407] Connection header received
0929 14:51:39.942960 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:39.942964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:39.943030 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:39.943118 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:39.944509 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.945222 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.946687 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.946934 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.947047 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:39.949885 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:39.949888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:39.949890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:39.949914 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:39.951559 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:39.952325 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:39.952328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:39.952330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:39.952415 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:39.952845 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:39.952847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:39.952848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:39.953053 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:39.953133 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:39.953852 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 14:51:39.953990 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:51:39.955490 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:39.940704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45665 (local address 127.0.98.1:43193)
0929 14:51:39.941225 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:39.941277 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:39.942712 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:39.943321 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:39.943328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:39.943759 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:39.944333 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:39.944347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.945349 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.945353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:39.946566 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:39.946575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:39.947089 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:39.947094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:39.947281 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:39.947977 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:39.947995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:39.949698 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:39.951711 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:39.951721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:39.951742 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:39.952164 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:39.952516 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:39.952519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:39.952523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:39.952735 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:39.953172 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:39.953190 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:39.953566 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 14:51:39.953800 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 14:51:39.954229 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":115,"threads_started":1}
I20260929 14:51:39.956908 19767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45665
I20260929 14:51:39.957266 19767 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:51:39.957837 19767 heartbeater.cc:511] Master 127.0.98.62:45665 requested a full tablet report, sending...
I20260929 14:51:39.959656 19576 ts_manager.cc:194] Registered new tserver with Master: cc4983714eb34a7bbbae0754697c0e4c (127.0.98.1:33453)
I20260929 14:51:39.961423 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:51:39.979012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:40.122208 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:40.122529 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:40.122578 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:40.126861 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:40.126940 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:40.127007 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:40.132805 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:40.134214 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:40.135499 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:40.147943 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:40.147852 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:40.147889 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:40.148358 19772 server_base.cc:1034] running on GCE node
I20260929 14:51:40.148978 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:40.149854 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:40.151075 19772 hybrid_clock.cc:648] HybridClock initialized: now 1790693500151022 us; error 69 us; skew 500 ppm
Sep 29 14:51:40 dist-test-slave-5hd1 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:40.154724 19772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:51:40.155870 19772 webserver.cc:460] Webserver started at http://127.0.98.2:41583/ using document root <none> and password file <none>
I20260929 14:51:40.156594 19772 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:40.156665 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:40.156873 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:40.159243 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "0951a657ad81201aedc153f72225c518"
server_key_iv: "1b47de504ab6449cbad54f67c8e4441a"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.159857 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "0951a657ad81201aedc153f72225c518"
server_key_iv: "1b47de504ab6449cbad54f67c8e4441a"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.164687 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 14:51:40.167876 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.169752 19772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:40.169893 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "0951a657ad81201aedc153f72225c518"
server_key_iv: "1b47de504ab6449cbad54f67c8e4441a"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.170012 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:40.185698 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:40.189240 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:40.189419 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:40.190079 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:40.190989 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:40.191037 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.191082 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:40.191097 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.202441 19772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40733
I20260929 14:51:40.202534 19900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40733 every 8 connection(s)
I20260929 14:51:40.203575 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:40.208398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19772
I20260929 14:51:40.208570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:40.208889 392 external_mini_cluster.cc:1428] Setting key 237b8c7d87ab0a30c7eb79dd080fef32
Sep 29 14:51:40 dist-test-slave-5hd1 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:51:40.221903 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.205896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47775 (local address 127.0.98.62:45665)
0929 14:51:40.206122 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.206127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.207236 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 14:51:40.208971 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.208978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.209052 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.209204 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:40.210719 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.211325 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.213046 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.213340 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.213462 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.217022 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:40.217030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:40.217035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:40.217073 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:40.219064 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.219740 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.219745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.219748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.219838 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.220314 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.220319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.220322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.220609 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:40.220697 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.221479 (+ 782us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.221661 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:51:40.223017 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.206431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45665 (local address 127.0.98.2:47775)
0929 14:51:40.207032 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.207085 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.208683 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.209410 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.209417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.209917 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:40.210534 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.210548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.211467 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.211474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.212896 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.212906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.213494 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.213498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.213671 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.214370 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:40.214395 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:40.216794 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:40.219221 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:40.219228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:40.219247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:40.219590 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:40.219948 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:40.219951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:40.219952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:40.220120 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:40.220748 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:40.220755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:40.221094 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.221363 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.221857 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":122,"threads_started":1}
I20260929 14:51:40.224550 19901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45665
I20260929 14:51:40.224874 19901 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:40.225513 19901 heartbeater.cc:511] Master 127.0.98.62:45665 requested a full tablet report, sending...
I20260929 14:51:40.226809 19576 ts_manager.cc:194] Registered new tserver with Master: a89a06d3e4eb4fdea8bebb5b29d0bef7 (127.0.98.2:40733)
I20260929 14:51:40.227576 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:51:40.239966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:40.374330 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:40.374648 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:40.374696 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:40.379221 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:40.379303 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:40.379370 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:40.385062 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:40.386406 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:40.387657 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:40.399297 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:40.399284 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:40.399449 19906 server_base.cc:1034] running on GCE node
W20260929 14:51:40.399284 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:40.400239 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:40.400995 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:40.402196 19906 hybrid_clock.cc:648] HybridClock initialized: now 1790693500402179 us; error 33 us; skew 500 ppm
Sep 29 14:51:40 dist-test-slave-5hd1 krb5kdc[19531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:40.405606 19906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:51:40.406605 19906 webserver.cc:460] Webserver started at http://127.0.98.3:37601/ using document root <none> and password file <none>
I20260929 14:51:40.407300 19906 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:40.407377 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:40.407619 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:40.410043 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "71e18cd2cc4849aa8af36d1611d90975"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "f85adcc7b3788effab3cbf0df238df19"
server_key_iv: "227676ec385ce7579823c6939fb3c9e9"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.410665 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71e18cd2cc4849aa8af36d1611d90975"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "f85adcc7b3788effab3cbf0df238df19"
server_key_iv: "227676ec385ce7579823c6939fb3c9e9"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.415004 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:51:40.417944 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.419474 19906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:40.419611 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "71e18cd2cc4849aa8af36d1611d90975"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "f85adcc7b3788effab3cbf0df238df19"
server_key_iv: "227676ec385ce7579823c6939fb3c9e9"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.419787 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:40.437276 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:40.441025 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:40.441264 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:40.441991 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:40.442950 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:40.443003 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.443045 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:40.443061 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.454020 19906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41337
I20260929 14:51:40.454109 20034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41337 every 8 connection(s)
I20260929 14:51:40.455093 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:40.457985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 19906
I20260929 14:51:40.458122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:40.458530 392 external_mini_cluster.cc:1428] Setting key d270f6ed9952a4d581169527d812f533
Sep 29 14:51:40 dist-test-slave-5hd1 krb5kdc[19531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:51:40.470953 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.457591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47625 (local address 127.0.98.62:45665)
0929 14:51:40.457821 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.457827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.458900 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 14:51:40.460750 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.460754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.460822 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.460925 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:40.462441 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.463041 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.464222 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.464472 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.464557 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.467341 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:40.467345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:40.467348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:40.467377 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:40.468813 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.469386 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.469388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.469389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.469461 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.469802 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.469803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.469805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.469999 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:40.470086 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.470694 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.470817 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 14:51:40.472150 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.458036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45665 (local address 127.0.98.3:47625)
0929 14:51:40.458700 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.458767 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.460475 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.461173 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.461181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.461778 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:40.462280 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.462293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.463154 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.463157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.464069 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.464078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.464762 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.464765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.464912 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.465557 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:51:40.465572 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:40.467147 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:40.468953 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:40.468959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:40.468973 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:40.469291 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:40.469554 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:51:40.469557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:51:40.469558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:51:40.469707 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:51:40.470124 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:40.470130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:40.470451 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.470641 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.470958 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":123,"threads_started":1}
I20260929 14:51:40.473592 20035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45665
I20260929 14:51:40.474000 20035 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:40.474617 20035 heartbeater.cc:511] Master 127.0.98.62:45665 requested a full tablet report, sending...
I20260929 14:51:40.475796 19576 ts_manager.cc:194] Registered new tserver with Master: 71e18cd2cc4849aa8af36d1611d90975 (127.0.98.3:41337)
I20260929 14:51:40.476500 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47625
I20260929 14:51:40.485154 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:40.495107 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.487442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.0.98.62:45665)
0929 14:51:40.487600 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.487604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.487799 (+ 195us) server_negotiation.cc:407] Connection header received
0929 14:51:40.487892 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.487894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.487942 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.488031 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:40.488801 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.489365 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.490207 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.490433 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.490516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.491853 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:40.491856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:40.491858 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:40.491880 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:40.493406 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.493863 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.493867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.493869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.493950 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:40.494258 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:40.494260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:40.494261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:40.494516 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:40.494631 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.494861 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.494968 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:51:40.498622 19576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:40.502539 19576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:40.520635 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.514096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33453 (local address 127.0.0.1:34272)
0929 14:51:40.514686 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.514702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.514804 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.515732 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.515737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.515759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:40.515860 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.515865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.516993 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.517000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.518986 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.518997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.519900 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.520023 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.520063 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.520269 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.520420 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":233,"thread_start_us":120,"threads_started":1}
I20260929 14:51:40.520625 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.514341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.0.98.1:33453)
0929 14:51:40.515155 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.515161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.515184 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:51:40.515271 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.515276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.515452 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.515595 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:40.515976 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.516871 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519126 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.519752 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.520066 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.520312 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.520429 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":107,"threads_started":1}
I20260929 14:51:40.520984 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.513228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41337 (local address 127.0.0.1:40066)
0929 14:51:40.514220 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.514286 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.514434 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.515235 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.515241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.515265 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:40.515437 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.515445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.517060 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.517063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.518921 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.518933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.519965 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.520107 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.520142 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.520226 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.520427 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":206,"thread_start_us":171,"threads_started":1}
I20260929 14:51:40.521240 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.513527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.2:40733)
0929 14:51:40.514342 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.514357 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.514759 (+ 402us) server_negotiation.cc:407] Connection header received
0929 14:51:40.514870 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.514878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.515134 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.515317 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:40.515876 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.517070 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.518900 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.519813 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519939 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.520103 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.520936 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.521061 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":148,"threads_started":1}
I20260929 14:51:40.521426 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.513228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.0.1:57366)
0929 14:51:40.514508 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.514524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.514651 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.515529 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.515536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.515558 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:40.515688 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.515699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.517307 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.517311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.518749 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.518760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.520162 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.520193 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.520283 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.520301 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.520368 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.520438 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":441,"thread_start_us":117,"threads_started":1}
I20260929 14:51:40.521803 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.513711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.98.3:41337)
0929 14:51:40.514705 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.514721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.514749 (+ 28us) server_negotiation.cc:407] Connection header received
0929 14:51:40.514836 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.514844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.515071 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.515242 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:40.515575 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.516861 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.519091 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.519818 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.520585 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.521246 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.521424 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.521625 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":182,"threads_started":1}
I20260929 14:51:40.524633 19969 tablet_service.cc:1467] Processing CreateTablet for tablet 90b249808ff64e5ca5d033717cfb885f (DEFAULT_TABLE table=test-table [id=cad9b8df5a3542f7a3cc373357441cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:40.524646 19701 tablet_service.cc:1467] Processing CreateTablet for tablet 90b249808ff64e5ca5d033717cfb885f (DEFAULT_TABLE table=test-table [id=cad9b8df5a3542f7a3cc373357441cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:40.525722 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b249808ff64e5ca5d033717cfb885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:40.526026 19701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b249808ff64e5ca5d033717cfb885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:40.527827 19835 tablet_service.cc:1467] Processing CreateTablet for tablet 90b249808ff64e5ca5d033717cfb885f (DEFAULT_TABLE table=test-table [id=cad9b8df5a3542f7a3cc373357441cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:40.529099 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b249808ff64e5ca5d033717cfb885f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:40.532761 20052 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Bootstrap starting.
I20260929 14:51:40.534477 20053 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Bootstrap starting.
I20260929 14:51:40.534945 20052 tablet_bootstrap.cc:654] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:40.536396 20052 log.cc:824] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:40.536866 20053 tablet_bootstrap.cc:654] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:40.537400 20054 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Bootstrap starting.
I20260929 14:51:40.537653 20053 log.cc:824] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:40.539521 20054 tablet_bootstrap.cc:654] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:40.539683 20053 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: No bootstrap required, opened a new log
I20260929 14:51:40.539700 20052 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: No bootstrap required, opened a new log
I20260929 14:51:40.539901 20052 ts_tablet_manager.cc:1397] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 14:51:40.540040 20053 ts_tablet_manager.cc:1397] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:40.540953 20054 log.cc:824] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:40.543300 20054 tablet_bootstrap.cc:492] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: No bootstrap required, opened a new log
I20260929 14:51:40.543514 20054 ts_tablet_manager.cc:1397] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:51:40.543928 20052 raft_consensus.cc:348] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.544235 20052 raft_consensus.cc:374] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:40.544286 20052 raft_consensus.cc:729] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e18cd2cc4849aa8af36d1611d90975, State: Initialized, Role: FOLLOWER
I20260929 14:51:40.544804 20052 consensus_queue.cc:260] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.544679 20053 raft_consensus.cc:348] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.545006 20053 raft_consensus.cc:374] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:40.545064 20053 raft_consensus.cc:729] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4983714eb34a7bbbae0754697c0e4c, State: Initialized, Role: FOLLOWER
I20260929 14:51:40.545631 20053 consensus_queue.cc:260] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.545861 20052 ts_tablet_manager.cc:1428] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:51:40.546233 20035 heartbeater.cc:503] Master 127.0.98.62:45665 was elected leader, sending a full tablet report...
I20260929 14:51:40.546453 19767 heartbeater.cc:503] Master 127.0.98.62:45665 was elected leader, sending a full tablet report...
I20260929 14:51:40.546837 20053 ts_tablet_manager.cc:1428] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:51:40.548668 20054 raft_consensus.cc:348] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.548925 20054 raft_consensus.cc:374] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:40.548988 20054 raft_consensus.cc:729] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89a06d3e4eb4fdea8bebb5b29d0bef7, State: Initialized, Role: FOLLOWER
I20260929 14:51:40.549460 20054 consensus_queue.cc:260] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.553368 20054 ts_tablet_manager.cc:1428] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260929 14:51:40.553830 19901 heartbeater.cc:503] Master 127.0.98.62:45665 was elected leader, sending a full tablet report...
I20260929 14:51:40.638111 20059 raft_consensus.cc:482] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:40.638424 20059 raft_consensus.cc:504] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.640194 20059 leader_election.cc:288] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71e18cd2cc4849aa8af36d1611d90975 (127.0.98.3:41337), a89a06d3e4eb4fdea8bebb5b29d0bef7 (127.0.98.2:40733)
I20260929 14:51:40.644737 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.640399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41337 (local address 127.0.98.1:33563)
0929 14:51:40.640884 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.640903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.641109 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.641537 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.641540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.641560 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:40.641742 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.641750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.642608 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.642614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.643497 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.643506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.644240 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.644243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.644354 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.644380 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.644484 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.644573 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":151,"threads_started":1}
I20260929 14:51:40.644719 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.640520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33563 (local address 127.0.98.3:41337)
0929 14:51:40.640700 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.640708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.641008 (+ 300us) server_negotiation.cc:407] Connection header received
0929 14:51:40.641247 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.641250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.641309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.641361 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:40.641887 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.642488 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.643635 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.644078 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.644235 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.644315 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.644501 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.644582 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 14:51:40.645423 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b249808ff64e5ca5d033717cfb885f" candidate_uuid: "cc4983714eb34a7bbbae0754697c0e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e18cd2cc4849aa8af36d1611d90975" is_pre_election: true
I20260929 14:51:40.645808 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.640476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.98.1:50501)
0929 14:51:40.641177 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.641192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:40.641332 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:40.641621 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:40.641624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:40.641640 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:40.641824 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.641834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.643074 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.643078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.644421 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:40.644436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.645335 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.645339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:40.645442 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.645462 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:51:40.645556 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 14:51:40.645644 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":178,"threads_started":1}
I20260929 14:51:40.645788 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.641084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50501 (local address 127.0.98.2:40733)
0929 14:51:40.641230 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.641236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.641258 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:51:40.641461 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.641470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.641516 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.641598 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:40.642000 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.642914 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.644591 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.645143 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.645222 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.645319 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.645572 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.645654 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:51:40.645814 19989 raft_consensus.cc:2454] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4983714eb34a7bbbae0754697c0e4c in term 0.
I20260929 14:51:40.646397 19657 leader_election.cc:302] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e18cd2cc4849aa8af36d1611d90975, cc4983714eb34a7bbbae0754697c0e4c; no voters:
I20260929 14:51:40.646576 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b249808ff64e5ca5d033717cfb885f" candidate_uuid: "cc4983714eb34a7bbbae0754697c0e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" is_pre_election: true
I20260929 14:51:40.646744 20059 raft_consensus.cc:2781] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:40.646862 20059 raft_consensus.cc:482] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:40.646917 20059 raft_consensus.cc:3037] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:40.646944 19855 raft_consensus.cc:2454] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4983714eb34a7bbbae0754697c0e4c in term 0.
I20260929 14:51:40.648079 20059 raft_consensus.cc:504] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.648507 20059 leader_election.cc:288] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [CANDIDATE]: Term 1 election: Requested vote from peers 71e18cd2cc4849aa8af36d1611d90975 (127.0.98.3:41337), a89a06d3e4eb4fdea8bebb5b29d0bef7 (127.0.98.2:40733)
I20260929 14:51:40.648986 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b249808ff64e5ca5d033717cfb885f" candidate_uuid: "cc4983714eb34a7bbbae0754697c0e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7"
I20260929 14:51:40.649000 19989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b249808ff64e5ca5d033717cfb885f" candidate_uuid: "cc4983714eb34a7bbbae0754697c0e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71e18cd2cc4849aa8af36d1611d90975"
I20260929 14:51:40.649111 19855 raft_consensus.cc:3037] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:40.649185 19989 raft_consensus.cc:3037] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:40.650445 19989 raft_consensus.cc:2454] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4983714eb34a7bbbae0754697c0e4c in term 1.
I20260929 14:51:40.650445 19855 raft_consensus.cc:2454] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4983714eb34a7bbbae0754697c0e4c in term 1.
I20260929 14:51:40.650821 19657 leader_election.cc:302] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e18cd2cc4849aa8af36d1611d90975, cc4983714eb34a7bbbae0754697c0e4c; no voters:
I20260929 14:51:40.651073 20059 raft_consensus.cc:2781] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:40.651335 20059 raft_consensus.cc:686] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 1 LEADER]: Becoming Leader. State: Replica: cc4983714eb34a7bbbae0754697c0e4c, State: Running, Role: LEADER
I20260929 14:51:40.651774 20059 consensus_queue.cc:237] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260929 14:51:40.655584 19575 catalog_manager.cc:5495] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4983714eb34a7bbbae0754697c0e4c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc4983714eb34a7bbbae0754697c0e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc4983714eb34a7bbbae0754697c0e4c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } health_report { overall_health: HEALTHY } } }
I20260929 14:51:40.676649 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.672133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.98.1:33453)
0929 14:51:40.672367 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.672372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.672395 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:51:40.672489 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.672492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.672564 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.672679 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:40.672951 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.673641 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.674811 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.675055 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.675139 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.675480 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:40.676048 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:40.676151 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.676360 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.676442 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 14:51:40.684048 19989 raft_consensus.cc:1261] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 1 FOLLOWER]: Refusing update from remote peer cc4983714eb34a7bbbae0754697c0e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:40.684073 19855 raft_consensus.cc:1261] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 1 FOLLOWER]: Refusing update from remote peer cc4983714eb34a7bbbae0754697c0e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:40.684901 20063 consensus_queue.cc:1035] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71e18cd2cc4849aa8af36d1611d90975" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:40.685237 20059 consensus_queue.cc:1035] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a89a06d3e4eb4fdea8bebb5b29d0bef7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:51:40.689617 19768 tablet.cc:2369] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:40.692981 20066 mvcc.cc:205] Tried to move back new op lower bound from 7334680578792472576 to 7334680578673311744. Current Snapshot: MvccSnapshot[applied={T|T < 7334680578792472576}]
I20260929 14:51:40.693794 20068 mvcc.cc:205] Tried to move back new op lower bound from 7334680578792472576 to 7334680578673311744. Current Snapshot: MvccSnapshot[applied={T|T < 7334680578792472576}]
I20260929 14:51:40.703601 19576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:40.703814 19576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:40.706984 19576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f897dbc9531941f2aa43aabd01ee2976: Sending DeleteTablet for 3 replicas of tablet 90b249808ff64e5ca5d033717cfb885f
I20260929 14:51:40.707858 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b249808ff64e5ca5d033717cfb885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40066
I20260929 14:51:40.707837 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b249808ff64e5ca5d033717cfb885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260929 14:51:40.708472 19701 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b249808ff64e5ca5d033717cfb885f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34272
I20260929 14:51:40.708983 20077 tablet_replica.cc:330] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c: stopping tablet replica
I20260929 14:51:40.709327 20077 raft_consensus.cc:2229] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:51:40.709688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19638
I20260929 14:51:40.709743 20077 raft_consensus.cc:2258] T 90b249808ff64e5ca5d033717cfb885f P cc4983714eb34a7bbbae0754697c0e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:40.711859 20076 tablet_replica.cc:330] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: stopping tablet replica
I20260929 14:51:40.712109 20076 raft_consensus.cc:2229] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:40.712455 20076 raft_consensus.cc:2258] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:40.714025 20076 ts_tablet_manager.cc:1905] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:51:40.716602 19562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33453 (error 108)
I20260929 14:51:40.716660 20075 tablet_replica.cc:330] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: stopping tablet replica
I20260929 14:51:40.716735 20076 ts_tablet_manager.cc:1918] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:40.716836 20076 log.cc:1196] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/90b249808ff64e5ca5d033717cfb885f
I20260929 14:51:40.717026 20075 raft_consensus.cc:2229] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:40.717165 20076 ts_tablet_manager.cc:1939] T 90b249808ff64e5ca5d033717cfb885f P a89a06d3e4eb4fdea8bebb5b29d0bef7: Deleting consensus metadata
I20260929 14:51:40.717406 20075 raft_consensus.cc:2258] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:40.717828 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.717540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33453 (local address 127.0.0.1:34282)
0929 14:51:40.717663 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.717739 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 14:51:40.717867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19772
W20260929 14:51:40.718679 19562 catalog_manager.cc:4584] TS cc4983714eb34a7bbbae0754697c0e4c (127.0.98.1:33453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90b249808ff64e5ca5d033717cfb885f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33453: connect: Connection refused (error 111)
I20260929 14:51:40.719877 20075 ts_tablet_manager.cc:1905] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:40.722193 20075 ts_tablet_manager.cc:1918] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:40.722311 20075 log.cc:1196] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/90b249808ff64e5ca5d033717cfb885f
I20260929 14:51:40.722605 20075 ts_tablet_manager.cc:1939] T 90b249808ff64e5ca5d033717cfb885f P 71e18cd2cc4849aa8af36d1611d90975: Deleting consensus metadata
I20260929 14:51:40.723830 19563 catalog_manager.cc:4841] TS 71e18cd2cc4849aa8af36d1611d90975 (127.0.98.3:41337): tablet 90b249808ff64e5ca5d033717cfb885f (table test-table [id=cad9b8df5a3542f7a3cc373357441cf4]) successfully deleted
I20260929 14:51:40.725454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19906
I20260929 14:51:40.726400 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.725530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40733 (local address 127.0.0.1:57376)
0929 14:51:40.726258 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:40.726309 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640}
I20260929 14:51:40.732692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 19544
2026-09-29T14:51:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2555 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T14:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:40Z Disabled control of system clock
I20260929 14:51:40.767597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35415
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:40.891888 20082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:40.892210 20082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:40.892248 20082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:40.896407 20082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:40.896473 20082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:40.896489 20082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:40.896509 20082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:40.896525 20082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:40.901335 20082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:40.902621 20082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:40.904307 20082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:40.914237 20088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:40.914245 20087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:40.914228 20090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:40.914443 20082 server_base.cc:1034] running on GCE node
I20260929 14:51:40.915405 20082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:40.916865 20082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:40.918089 20082 hybrid_clock.cc:648] HybridClock initialized: now 1790693500918041 us; error 59 us; skew 500 ppm
I20260929 14:51:40.926905 20082 webserver.cc:460] Webserver started at http://127.0.98.62:46475/ using document root <none> and password file <none>
I20260929 14:51:40.927676 20082 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:40.927752 20082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:40.928002 20082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:40.930316 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "08aef8f83fd82f896d71b6345ce24755"
server_key_iv: "9debd0bdcdee3cfb600e9b7af0fb4f36"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.930922 20082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "08aef8f83fd82f896d71b6345ce24755"
server_key_iv: "9debd0bdcdee3cfb600e9b7af0fb4f36"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.935199 20082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:51:40.938112 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:40.939706 20082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:51:40.939878 20082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a"
format_stamp: "Formatted at 2026-09-29 14:51:40 on dist-test-slave-5hd1"
server_key: "08aef8f83fd82f896d71b6345ce24755"
server_key_iv: "9debd0bdcdee3cfb600e9b7af0fb4f36"
server_key_version: "encryptionkey@0"
I20260929 14:51:40.940017 20082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:40.974133 20082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:40.974912 20082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:40.975147 20082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:40.982728 20082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35415
I20260929 14:51:40.982816 20148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35415 every 8 connection(s)
I20260929 14:51:40.983750 20082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:40.985798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20082
I20260929 14:51:40.985975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:40.986356 392 external_mini_cluster.cc:1428] Setting key 2284d2d215f205a3475b9c1e76c86d7f
I20260929 14:51:40.988119 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:40.995981 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: Bootstrap starting.
I20260929 14:51:40.997783 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:40.987953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.0.98.62:35415)
0929 14:51:40.988729 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:40.988753 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:40.988800 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:51:40.989750 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:40.989775 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:40.990323 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:40.990723 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:40.991235 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.992799 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.993718 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:40.994433 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:40.994611 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:40.995477 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:40.995483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:40.995500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:40.996052 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:40.996128 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:40.996154 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:40.996284 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:40.996445 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:51:40.996747 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":180,"threads_started":1}
I20260929 14:51:40.998996 20149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:40.999980 20149 log.cc:824] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:41.002477 20149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: No bootstrap required, opened a new log
I20260929 14:51:41.006368 20149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } }
I20260929 14:51:41.006616 20149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:41.006688 20149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc5dd1fcde44b2db1a538a8b4ab579a, State: Initialized, Role: FOLLOWER
I20260929 14:51:41.007197 20149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } }
I20260929 14:51:41.007341 20149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:41.007431 20149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:41.007558 20149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:41.008675 20149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } }
I20260929 14:51:41.009054 20149 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc5dd1fcde44b2db1a538a8b4ab579a; no voters:
I20260929 14:51:41.009469 20149 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:41.009559 20154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:41.009747 20154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [term 1 LEADER]: Becoming Leader. State: Replica: 9dc5dd1fcde44b2db1a538a8b4ab579a, State: Running, Role: LEADER
I20260929 14:51:41.010067 20154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } }
I20260929 14:51:41.010895 20149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:41.011845 20156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } } }
I20260929 14:51:41.011987 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc5dd1fcde44b2db1a538a8b4ab579a. Latest consensus state: current_term: 1 leader_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5dd1fcde44b2db1a538a8b4ab579a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35415 } } }
I20260929 14:51:41.012064 20156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:41.012106 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:41.012601 20163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:41.015805 20163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:41.021919 20163 catalog_manager.cc:1347] Generated new cluster ID: 0671798961a0436eb42b30ea70f798c5
I20260929 14:51:41.021996 20163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:41.027343 20163 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:41.028709 20163 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:41.037192 20163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: Generated new TSK 0
I20260929 14:51:41.038002 20163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:41.047827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:51:41.177446 20173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:41.177766 20173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:41.177809 20173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:41.181998 20173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:41.182073 20173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:41.182133 20173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:51:41.187192 20173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:41.188552 20173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:41.190205 20173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:41.201525 20181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:41.201622 20178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:41.201555 20179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:41.201915 20173 server_base.cc:1034] running on GCE node
I20260929 14:51:41.202455 20173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:41.203212 20173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:41.204408 20173 hybrid_clock.cc:648] HybridClock initialized: now 1790693501204388 us; error 39 us; skew 500 ppm
I20260929 14:51:41.206514 20173 webserver.cc:460] Webserver started at http://127.0.98.1:43049/ using document root <none> and password file <none>
I20260929 14:51:41.207137 20173 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:41.207204 20173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:41.207438 20173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:41.209728 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c6784cef1b945eca136c6d60685f774"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "34ee57303f19b9483307664daf7d80b5"
server_key_iv: "da0762c1d36ee30ac7ff486ac9d3ea00"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.210328 20173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c6784cef1b945eca136c6d60685f774"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "34ee57303f19b9483307664daf7d80b5"
server_key_iv: "da0762c1d36ee30ac7ff486ac9d3ea00"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.214571 20173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:51:41.217319 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.218811 20173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:41.218982 20173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal
uuid: "3c6784cef1b945eca136c6d60685f774"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "34ee57303f19b9483307664daf7d80b5"
server_key_iv: "da0762c1d36ee30ac7ff486ac9d3ea00"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.219115 20173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:41.233716 20173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:41.234483 20173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:41.234701 20173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:41.235345 20173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:41.236464 20173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:41.236536 20173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.236606 20173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:41.236649 20173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.247031 20173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36481
I20260929 14:51:41.247126 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36481 every 8 connection(s)
I20260929 14:51:41.248040 20173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:51:41.255834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20173
I20260929 14:51:41.255981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:51:41.256383 392 external_mini_cluster.cc:1428] Setting key 1ec47d1a15339362192d4c678557aa9f
I20260929 14:51:41.258739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:41.259810 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.250524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43009 (local address 127.0.98.62:35415)
0929 14:51:41.250716 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.250721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.251961 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 14:51:41.253718 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.253722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.253797 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.253921 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:41.255153 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.255980 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.257441 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.257668 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.257783 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.259224 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:41.259228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:41.259230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:41.259245 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:41.259265 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:41.259273 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:41.259377 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.259563 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.259654 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:51:41.261045 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.250850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35415 (local address 127.0.98.1:43009)
0929 14:51:41.251732 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.251807 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.253443 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.254129 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.254137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.254241 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:41.254914 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.254930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.256219 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.256227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.257293 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.257307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.257834 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.257838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.258016 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.258708 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:41.258709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:41.258754 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:41.258864 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:41.259379 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:41.259392 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:41.259400 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.259564 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.259877 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":176,"threads_started":1}
I20260929 14:51:41.262733 20301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35415
I20260929 14:51:41.263015 20301 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:41.263599 20301 heartbeater.cc:511] Master 127.0.98.62:35415 requested a full tablet report, sending...
I20260929 14:51:41.265249 20113 ts_manager.cc:194] Registered new tserver with Master: 3c6784cef1b945eca136c6d60685f774 (127.0.98.1:36481)
I20260929 14:51:41.266394 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43009
W20260929 14:51:41.382686 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:41.383056 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:41.383145 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:41.387540 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:41.387637 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:41.387742 20304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:51:41.392849 20304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:41.394029 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:41.395462 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:41.406229 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:41.406245 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:41.406420 20304 server_base.cc:1034] running on GCE node
W20260929 14:51:41.406283 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:41.407085 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:41.407763 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:41.408974 20304 hybrid_clock.cc:648] HybridClock initialized: now 1790693501408939 us; error 48 us; skew 500 ppm
I20260929 14:51:41.410928 20304 webserver.cc:460] Webserver started at http://127.0.98.2:38929/ using document root <none> and password file <none>
I20260929 14:51:41.411530 20304 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:41.411625 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:41.411832 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:41.414063 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f49d597039b411a8d9a129e18b6f851"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "6f1fa4ca035f2325317d155d2bcce1bf"
server_key_iv: "f9d01a893080adc1dfd45843d0ac4819"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.414610 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f49d597039b411a8d9a129e18b6f851"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "6f1fa4ca035f2325317d155d2bcce1bf"
server_key_iv: "f9d01a893080adc1dfd45843d0ac4819"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.418468 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:51:41.421213 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.422680 20304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:41.422824 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal
uuid: "7f49d597039b411a8d9a129e18b6f851"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "6f1fa4ca035f2325317d155d2bcce1bf"
server_key_iv: "f9d01a893080adc1dfd45843d0ac4819"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.422943 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:41.437322 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:41.438248 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:41.438495 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:41.439162 20304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:41.440264 20304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:41.440348 20304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.440420 20304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:41.440476 20304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.451874 20304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34921
I20260929 14:51:41.451897 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34921 every 8 connection(s)
I20260929 14:51:41.452960 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:51:41.457535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20304
I20260929 14:51:41.457708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:51:41.458076 392 external_mini_cluster.cc:1428] Setting key 45358ee02975090f1b573f7701e6cb95
I20260929 14:51:41.460374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--builtin_ntp_servers=127.0.98.20:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:51:41.465103 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.455432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47171 (local address 127.0.98.62:35415)
0929 14:51:41.455612 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.455617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.456938 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 14:51:41.458950 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.458956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.459025 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.459138 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:41.460419 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.461291 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.462706 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.462974 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.463099 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.464506 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:41.464510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:41.464515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:41.464537 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:41.464558 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:41.464568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:41.464654 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.464843 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.464923 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:51:41.466157 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.455835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35415 (local address 127.0.98.2:47171)
0929 14:51:41.456743 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.456797 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.458549 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.459351 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.459359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.459496 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:41.460251 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.460276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.461461 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.461465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.462548 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.462560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.463159 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.463163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.463374 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.464146 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:41.464147 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:41.464246 (+ 99us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:41.464338 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:41.464655 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:41.464672 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:41.464682 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.464829 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.465167 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":131,"threads_started":1}
I20260929 14:51:41.467748 20432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35415
I20260929 14:51:41.468046 20432 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:41.468735 20432 heartbeater.cc:511] Master 127.0.98.62:35415 requested a full tablet report, sending...
I20260929 14:51:41.470099 20113 ts_manager.cc:194] Registered new tserver with Master: 7f49d597039b411a8d9a129e18b6f851 (127.0.98.2:34921)
I20260929 14:51:41.471108 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47171
W20260929 14:51:41.575484 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:41.575847 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:41.575922 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:41.580325 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:41.580408 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:41.580467 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:51:41.585628 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:41.586824 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:41.588195 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:41.600455 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:41.600577 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:41.600483 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:41.600914 20435 server_base.cc:1034] running on GCE node
I20260929 14:51:41.601418 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:41.602272 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:41.603433 20435 hybrid_clock.cc:648] HybridClock initialized: now 1790693501603378 us; error 69 us; skew 500 ppm
I20260929 14:51:41.605400 20435 webserver.cc:460] Webserver started at http://127.0.98.3:34995/ using document root <none> and password file <none>
I20260929 14:51:41.606001 20435 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:41.606066 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:41.606287 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:41.608440 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ff5152cf66644b8a0dfec3e517d472b"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "2769da4c0902f6c0efaf2c35eb37849b"
server_key_iv: "f1ba3160a748fa73d3e3367b67cf59a0"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.609050 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff5152cf66644b8a0dfec3e517d472b"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "2769da4c0902f6c0efaf2c35eb37849b"
server_key_iv: "f1ba3160a748fa73d3e3367b67cf59a0"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.612987 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:51:41.615768 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.617331 20435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:41.617563 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal
uuid: "6ff5152cf66644b8a0dfec3e517d472b"
format_stamp: "Formatted at 2026-09-29 14:51:41 on dist-test-slave-5hd1"
server_key: "2769da4c0902f6c0efaf2c35eb37849b"
server_key_iv: "f1ba3160a748fa73d3e3367b67cf59a0"
server_key_version: "encryptionkey@0"
I20260929 14:51:41.617730 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:41.635198 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:41.635950 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:41.636165 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:41.636868 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:51:41.638113 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:51:41.638172 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.638243 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:51:41.638280 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:41.650058 20435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45483
I20260929 14:51:41.650099 20562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45483 every 8 connection(s)
I20260929 14:51:41.651154 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:51:41.660046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20435
I20260929 14:51:41.660221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:51:41.660529 392 external_mini_cluster.cc:1428] Setting key 0d43f0662328dceac585061fc11daeb1
I20260929 14:51:41.662962 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.653784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50163 (local address 127.0.98.62:35415)
0929 14:51:41.653990 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.653995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.655213 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 14:51:41.656912 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.656916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.656999 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.657108 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:41.658233 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.658829 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.660137 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.660471 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.660588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.662217 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:41.662221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:41.662225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:41.662245 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:41.662278 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:41.662287 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:41.662399 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.662649 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.662733 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:51:41.663982 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.654117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35415 (local address 127.0.98.3:50163)
0929 14:51:41.655011 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.655063 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.656651 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.657334 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.657342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.657447 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:51:41.658061 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.658075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.658967 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.658973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.659995 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.660005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.660659 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.660663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.660868 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.661735 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:51:41.661737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:51:41.661817 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:51:41.661930 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:51:41.662440 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:51:41.662452 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:51:41.662466 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.662611 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.662931 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":124,"threads_started":1}
I20260929 14:51:41.665411 20563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35415
I20260929 14:51:41.665732 20563 heartbeater.cc:464] Registering TS with master...
I20260929 14:51:41.666414 20563 heartbeater.cc:511] Master 127.0.98.62:35415 requested a full tablet report, sending...
I20260929 14:51:41.667649 20113 ts_manager.cc:194] Registered new tserver with Master: 6ff5152cf66644b8a0dfec3e517d472b (127.0.98.3:45483)
I20260929 14:51:41.668355 20113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50163
I20260929 14:51:41.675673 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:51:41.681839 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.678028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.98.62:35415)
0929 14:51:41.678181 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.678185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.678314 (+ 129us) server_negotiation.cc:407] Connection header received
0929 14:51:41.678426 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.678429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.678479 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.678557 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:41.678842 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.679405 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.680342 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.680556 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.680637 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.681439 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:41.681442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:41.681444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:41.681456 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:41.681475 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:41.681483 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:41.681545 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.681629 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.681676 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:51:41.685313 20113 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:51:41.688549 20113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:51:41.705950 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.699882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.98.1:36481)
0929 14:51:41.700507 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.700525 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.700553 (+ 28us) server_negotiation.cc:407] Connection header received
0929 14:51:41.700642 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.700646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.700860 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.701032 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:41.701382 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702701 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.704021 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.704859 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.704968 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.705216 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.705620 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.705741 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":164,"threads_started":1}
I20260929 14:51:41.705945 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.698961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36481 (local address 127.0.0.1:38726)
0929 14:51:41.700364 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.700380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.700471 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.701056 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.701059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.701076 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:41.701230 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.701240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702875 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.702877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.703879 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.703887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.705278 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.705281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.705409 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.705450 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.705582 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.705743 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260929 14:51:41.706228 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.698930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34921 (local address 127.0.0.1:39840)
0929 14:51:41.699912 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.699939 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.700110 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.701254 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.701257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.701275 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:41.701377 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.701383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702827 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.702830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.704407 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.704419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.705729 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.705733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.705834 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.705863 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.705919 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.706027 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":102,"thread_start_us":144,"threads_started":1}
I20260929 14:51:41.706444 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.698802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45483 (local address 127.0.0.1:56526)
0929 14:51:41.699870 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.699934 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.700117 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.700855 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.700862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.700888 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:41.701077 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.701086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702632 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.702639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.704554 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.704564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.705997 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.706001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.706100 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.706122 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.706196 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.706279 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":149,"spinlock_wait_cycles":1920,"thread_start_us":107,"threads_started":1}
I20260929 14:51:41.707026 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.698998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.98.3:45483)
0929 14:51:41.699571 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.699589 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.700099 (+ 510us) server_negotiation.cc:407] Connection header received
0929 14:51:41.700380 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.700387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.700656 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.700850 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:41.701238 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702509 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.704691 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.705572 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.705703 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.705927 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.706726 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.706853 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":185,"threads_started":1}
I20260929 14:51:41.707123 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.700363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.98.2:34921)
0929 14:51:41.700825 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.700837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.700863 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:51:41.700944 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.700948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.701112 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.701277 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:41.701512 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.702678 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.704599 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.705572 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.706576 (+ 1004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.706789 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.706839 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.706935 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":117,"threads_started":1}
I20260929 14:51:41.710055 20497 tablet_service.cc:1467] Processing CreateTablet for tablet dc5917711e32476d8b3f254b93bf8d45 (DEFAULT_TABLE table=test-table [id=867742d6bc3344f992520b8cb35e3998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:41.710253 20235 tablet_service.cc:1467] Processing CreateTablet for tablet dc5917711e32476d8b3f254b93bf8d45 (DEFAULT_TABLE table=test-table [id=867742d6bc3344f992520b8cb35e3998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:41.710253 20366 tablet_service.cc:1467] Processing CreateTablet for tablet dc5917711e32476d8b3f254b93bf8d45 (DEFAULT_TABLE table=test-table [id=867742d6bc3344f992520b8cb35e3998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:51:41.711251 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5917711e32476d8b3f254b93bf8d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:41.711510 20235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5917711e32476d8b3f254b93bf8d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:41.711829 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc5917711e32476d8b3f254b93bf8d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:41.718605 20580 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Bootstrap starting.
I20260929 14:51:41.720491 20582 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Bootstrap starting.
I20260929 14:51:41.720491 20581 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Bootstrap starting.
I20260929 14:51:41.722213 20580 tablet_bootstrap.cc:654] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:41.723021 20582 tablet_bootstrap.cc:654] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:41.723021 20581 tablet_bootstrap.cc:654] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:41.723109 20580 log.cc:824] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:41.723959 20581 log.cc:824] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:41.723959 20582 log.cc:824] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:41.725343 20580 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: No bootstrap required, opened a new log
I20260929 14:51:41.725584 20580 ts_tablet_manager.cc:1397] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:51:41.726199 20581 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: No bootstrap required, opened a new log
I20260929 14:51:41.726209 20582 tablet_bootstrap.cc:492] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: No bootstrap required, opened a new log
I20260929 14:51:41.726405 20581 ts_tablet_manager.cc:1397] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:51:41.726405 20582 ts_tablet_manager.cc:1397] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:51:41.730090 20580 raft_consensus.cc:348] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.730376 20580 raft_consensus.cc:374] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:41.730440 20580 raft_consensus.cc:729] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff5152cf66644b8a0dfec3e517d472b, State: Initialized, Role: FOLLOWER
I20260929 14:51:41.730932 20580 consensus_queue.cc:260] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.731081 20582 raft_consensus.cc:348] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.731081 20581 raft_consensus.cc:348] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.731369 20582 raft_consensus.cc:374] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:41.731369 20581 raft_consensus.cc:374] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:41.731437 20582 raft_consensus.cc:729] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f49d597039b411a8d9a129e18b6f851, State: Initialized, Role: FOLLOWER
I20260929 14:51:41.731444 20581 raft_consensus.cc:729] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6784cef1b945eca136c6d60685f774, State: Initialized, Role: FOLLOWER
I20260929 14:51:41.731874 20563 heartbeater.cc:503] Master 127.0.98.62:35415 was elected leader, sending a full tablet report...
I20260929 14:51:41.731983 20582 consensus_queue.cc:260] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.731983 20581 consensus_queue.cc:260] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.732120 20580 ts_tablet_manager.cc:1428] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:51:41.733068 20582 ts_tablet_manager.cc:1428] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260929 14:51:41.733104 20581 ts_tablet_manager.cc:1428] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:51:41.733456 20432 heartbeater.cc:503] Master 127.0.98.62:35415 was elected leader, sending a full tablet report...
I20260929 14:51:41.733456 20301 heartbeater.cc:503] Master 127.0.98.62:35415 was elected leader, sending a full tablet report...
I20260929 14:51:41.734028 20587 raft_consensus.cc:482] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:51:41.734314 20587 raft_consensus.cc:504] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.735846 20587 leader_election.cc:288] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff5152cf66644b8a0dfec3e517d472b (127.0.98.3:45483), 7f49d597039b411a8d9a129e18b6f851 (127.0.98.2:34921)
I20260929 14:51:41.741871 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.736119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45483 (local address 127.0.98.1:55439)
0929 14:51:41.736391 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.736410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.736502 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.737275 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.737279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.737298 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:41.737457 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.737466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.738654 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.738657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.740368 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.740380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.741399 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.741403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.741552 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.741584 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.741677 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.741756 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":94}
I20260929 14:51:41.742311 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.736787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55439 (local address 127.0.98.3:45483)
0929 14:51:41.736919 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.736925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.736940 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:51:41.736989 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.736992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.737043 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.737122 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:41.737615 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.738495 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.740536 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.741256 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.741373 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.741449 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.742093 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.742221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:51:41.743021 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.736370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33201 (local address 127.0.98.2:34921)
0929 14:51:41.736518 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.736524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.736774 (+ 250us) server_negotiation.cc:407] Connection header received
0929 14:51:41.737037 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.737041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.737096 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.737152 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:51:41.737655 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.738495 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.740536 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.741239 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.741373 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.741449 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.742834 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.742921 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:51:41.743115 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5917711e32476d8b3f254b93bf8d45" candidate_uuid: "3c6784cef1b945eca136c6d60685f774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff5152cf66644b8a0dfec3e517d472b" is_pre_election: true
I20260929 14:51:41.743412 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.736247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34921 (local address 127.0.98.1:33201)
0929 14:51:41.736684 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.736700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:51:41.736879 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:51:41.737319 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:51:41.737322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:51:41.737350 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:51:41.737488 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.737495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.738654 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.738657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.740368 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:51:41.740380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.742539 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.742543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:51:41.742645 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.742677 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 14:51:41.743235 (+ 558us) client_negotiation.cc:240] Negotiation successful
0929 14:51:41.743299 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":157,"spinlock_wait_cycles":8576,"thread_start_us":117,"threads_started":1}
I20260929 14:51:41.743604 20517 raft_consensus.cc:2454] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6784cef1b945eca136c6d60685f774 in term 0.
I20260929 14:51:41.744349 20191 leader_election.cc:302] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6784cef1b945eca136c6d60685f774, 6ff5152cf66644b8a0dfec3e517d472b; no voters:
I20260929 14:51:41.744402 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5917711e32476d8b3f254b93bf8d45" candidate_uuid: "3c6784cef1b945eca136c6d60685f774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f49d597039b411a8d9a129e18b6f851" is_pre_election: true
I20260929 14:51:41.744671 20587 raft_consensus.cc:2781] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:51:41.744773 20587 raft_consensus.cc:482] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:51:41.744748 20386 raft_consensus.cc:2454] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6784cef1b945eca136c6d60685f774 in term 0.
I20260929 14:51:41.744815 20587 raft_consensus.cc:3037] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:41.746233 20587 raft_consensus.cc:504] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
I20260929 14:51:41.746676 20587 leader_election.cc:288] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [CANDIDATE]: Term 1 election: Requested vote from peers 6ff5152cf66644b8a0dfec3e517d472b (127.0.98.3:45483), 7f49d597039b411a8d9a129e18b6f851 (127.0.98.2:34921)
I20260929 14:51:41.746994 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5917711e32476d8b3f254b93bf8d45" candidate_uuid: "3c6784cef1b945eca136c6d60685f774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff5152cf66644b8a0dfec3e517d472b"
I20260929 14:51:41.747133 20517 raft_consensus.cc:3037] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:41.747165 20386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc5917711e32476d8b3f254b93bf8d45" candidate_uuid: "3c6784cef1b945eca136c6d60685f774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f49d597039b411a8d9a129e18b6f851"
I20260929 14:51:41.747325 20386 raft_consensus.cc:3037] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:41.748447 20517 raft_consensus.cc:2454] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6784cef1b945eca136c6d60685f774 in term 1.
I20260929 14:51:41.748747 20191 leader_election.cc:302] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6784cef1b945eca136c6d60685f774, 6ff5152cf66644b8a0dfec3e517d472b; no voters:
I20260929 14:51:41.748766 20386 raft_consensus.cc:2454] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6784cef1b945eca136c6d60685f774 in term 1.
I20260929 14:51:41.748934 20587 raft_consensus.cc:2781] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:41.749044 20587 raft_consensus.cc:686] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6784cef1b945eca136c6d60685f774, State: Running, Role: LEADER
I20260929 14:51:41.749424 20587 consensus_queue.cc:237] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } }
W20260929 14:51:41.749538 20302 tablet.cc:2369] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:51:41.753319 20112 catalog_manager.cc:5495] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6784cef1b945eca136c6d60685f774 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c6784cef1b945eca136c6d60685f774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6784cef1b945eca136c6d60685f774" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 } health_report { overall_health: UNKNOWN } } }
I20260929 14:51:41.775959 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.771862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.0.98.1:36481)
0929 14:51:41.772023 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:41.772028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:41.772044 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:51:41.772145 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:41.772148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:41.772286 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:41.772398 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:51:41.772681 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.773230 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.774187 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:41.774409 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:41.774510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:41.774862 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:51:41.775402 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:51:41.775533 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:41.775678 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 14:51:41.775737 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:51:41.783207 20386 raft_consensus.cc:1261] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 1 FOLLOWER]: Refusing update from remote peer 3c6784cef1b945eca136c6d60685f774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:41.783262 20517 raft_consensus.cc:1261] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 1 FOLLOWER]: Refusing update from remote peer 3c6784cef1b945eca136c6d60685f774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:51:41.784125 20587 consensus_queue.cc:1035] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f49d597039b411a8d9a129e18b6f851" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:41.784402 20588 consensus_queue.cc:1035] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff5152cf66644b8a0dfec3e517d472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:51:41.793286 20595 mvcc.cc:205] Tried to move back new op lower bound from 7334680583295123456 to 7334680583169470464. Current Snapshot: MvccSnapshot[applied={T|T < 7334680583295123456}]
I20260929 14:51:41.803574 20112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:41.803839 20112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:51:41.806854 20112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dc5dd1fcde44b2db1a538a8b4ab579a: Sending DeleteTablet for 3 replicas of tablet dc5917711e32476d8b3f254b93bf8d45
I20260929 14:51:41.807825 20235 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5917711e32476d8b3f254b93bf8d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:41 UTC) from {username='slave'} at 127.0.0.1:38726
I20260929 14:51:41.807883 20366 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5917711e32476d8b3f254b93bf8d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:41 UTC) from {username='slave'} at 127.0.0.1:39840
I20260929 14:51:41.808313 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet dc5917711e32476d8b3f254b93bf8d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:51:41 UTC) from {username='slave'} at 127.0.0.1:56526
I20260929 14:51:41.808485 20603 tablet_replica.cc:330] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: stopping tablet replica
I20260929 14:51:41.808859 20603 raft_consensus.cc:2229] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:41.808928 20605 tablet_replica.cc:330] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: stopping tablet replica
I20260929 14:51:41.809206 20605 raft_consensus.cc:2229] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:51:41.809262 20603 raft_consensus.cc:2258] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:41.809520 20605 raft_consensus.cc:2258] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:51:41.809654 20604 tablet_replica.cc:330] T dc5917711e32476d8b3f254b93bf8d45 P 3c6784cef1b945eca136c6d60685f774: stopping tablet replica
I20260929 14:51:41.809811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20173
I20260929 14:51:41.812350 20603 ts_tablet_manager.cc:1905] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:41.814040 20605 ts_tablet_manager.cc:1905] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:51:41.815627 20603 ts_tablet_manager.cc:1918] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:41.815747 20603 log.cc:1196] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-1/wal/wals/dc5917711e32476d8b3f254b93bf8d45
I20260929 14:51:41.816084 20603 ts_tablet_manager.cc:1939] T dc5917711e32476d8b3f254b93bf8d45 P 7f49d597039b411a8d9a129e18b6f851: Deleting consensus metadata
I20260929 14:51:41.817057 20605 ts_tablet_manager.cc:1918] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:51:41.817157 20605 log.cc:1196] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Deleting WAL directory at /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790693238534145-392-0/minicluster-data/ts-2/wal/wals/dc5917711e32476d8b3f254b93bf8d45
I20260929 14:51:41.817459 20605 ts_tablet_manager.cc:1939] T dc5917711e32476d8b3f254b93bf8d45 P 6ff5152cf66644b8a0dfec3e517d472b: Deleting consensus metadata
I20260929 14:51:41.817747 20098 catalog_manager.cc:4841] TS 7f49d597039b411a8d9a129e18b6f851 (127.0.98.2:34921): tablet dc5917711e32476d8b3f254b93bf8d45 (table test-table [id=867742d6bc3344f992520b8cb35e3998]) successfully deleted
W20260929 14:51:41.817879 20099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36481 (error 108)
I20260929 14:51:41.817986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20304
I20260929 14:51:41.819658 20100 catalog_manager.cc:4841] TS 6ff5152cf66644b8a0dfec3e517d472b (127.0.98.3:45483): tablet dc5917711e32476d8b3f254b93bf8d45 (table test-table [id=867742d6bc3344f992520b8cb35e3998]) successfully deleted
I20260929 14:51:41.820427 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:41.819953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36481 (local address 127.0.0.1:38734)
0929 14:51:41.820151 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 14:51:41.820301 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 14:51:41.820600 20099 catalog_manager.cc:4584] TS 3c6784cef1b945eca136c6d60685f774 (127.0.98.1:36481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc5917711e32476d8b3f254b93bf8d45: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36481: connect: Connection refused (error 111)
I20260929 14:51:41.825356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20435
I20260929 14:51:41.832139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20082
2026-09-29T14:51:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1097 ms)
[----------] 6 tests from EncryptionPolicyTest (11188 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:41 dist-test-slave-5hd1 krb5kdc[20609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:41 dist-test-slave-5hd1 krb5kdc[20609](info): set up 2 sockets
Sep 29 14:51:41 dist-test-slave-5hd1 krb5kdc[20609](info): commencing operation
krb5kdc: starting...
W20260929 14:51:42.880600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:42 dist-test-slave-5hd1 krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:43.019779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:43.131512 20622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:43.131850 20622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:43.131932 20622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:43.136124 20622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:43.136237 20622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:43.136277 20622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:43.136297 20622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:43.136317 20622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:43.142076 20622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:43.143072 20622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:43.144071 20622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:43.152961 20628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:43.152961 20630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:43.152997 20627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:43.153335 20622 server_base.cc:1034] running on GCE node
I20260929 14:51:43.154199 20622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:43.155624 20622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:43.156823 20622 hybrid_clock.cc:648] HybridClock initialized: now 1790693503156811 us; error 47 us; skew 500 ppm
Sep 29 14:51:43 dist-test-slave-5hd1 krb5kdc[20609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:43.160068 20622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:43.161156 20622 webserver.cc:460] Webserver started at http://127.0.0.1:36665/ using document root <none> and password file <none>
I20260929 14:51:43.161856 20622 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:43.161957 20622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:43.162216 20622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:43.164646 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "274337de94a840c09796f28ff1d3b6b4"
format_stamp: "Formatted at 2026-09-29 14:51:43 on dist-test-slave-5hd1"
server_key: "b64b60a59e0a519294fd8b3ff223b857"
server_key_iv: "260f33346d7265e6ee1ee33a064583b1"
server_key_version: "encryptionkey@0"
I20260929 14:51:43.165256 20622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "274337de94a840c09796f28ff1d3b6b4"
format_stamp: "Formatted at 2026-09-29 14:51:43 on dist-test-slave-5hd1"
server_key: "b64b60a59e0a519294fd8b3ff223b857"
server_key_iv: "260f33346d7265e6ee1ee33a064583b1"
server_key_version: "encryptionkey@0"
I20260929 14:51:43.169572 20622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 14:51:43.172539 20637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:43.174005 20622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:51:43.174160 20622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "274337de94a840c09796f28ff1d3b6b4"
format_stamp: "Formatted at 2026-09-29 14:51:43 on dist-test-slave-5hd1"
server_key: "b64b60a59e0a519294fd8b3ff223b857"
server_key_iv: "260f33346d7265e6ee1ee33a064583b1"
server_key_version: "encryptionkey@0"
I20260929 14:51:43.174284 20622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:43.186352 20622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:43.189981 20622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:43.190214 20622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:43.199308 20622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35611
I20260929 14:51:43.199337 20689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35611 every 8 connection(s)
I20260929 14:51:43.200584 20622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:43.203613 20690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:43.207885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20622
I20260929 14:51:43.208068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:43.208431 392 external_mini_cluster.cc:1428] Setting key 9c614a8fb4207bb8bed7a115d809927d
I20260929 14:51:43.209630 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4: Bootstrap starting.
I20260929 14:51:43.211939 20690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:43.212750 20690 log.cc:824] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:43.214625 20690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4: No bootstrap required, opened a new log
Sep 29 14:51:43 dist-test-slave-5hd1 krb5kdc[20609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:51:43.218631 20690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } }
I20260929 14:51:43.218931 20690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:43.219118 20690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274337de94a840c09796f28ff1d3b6b4, State: Initialized, Role: FOLLOWER
I20260929 14:51:43.219719 20690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } }
I20260929 14:51:43.219874 20690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:43.219978 20690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:43.220106 20690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:43.221434 20690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } }
I20260929 14:51:43.221853 20690 leader_election.cc:302] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274337de94a840c09796f28ff1d3b6b4; no voters:
I20260929 14:51:43.222249 20690 leader_election.cc:288] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:43.222335 20695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:43.222582 20695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [term 1 LEADER]: Becoming Leader. State: Replica: 274337de94a840c09796f28ff1d3b6b4, State: Running, Role: LEADER
I20260929 14:51:43.222931 20695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } }
I20260929 14:51:43.223672 20690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:43.224978 20697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274337de94a840c09796f28ff1d3b6b4. Latest consensus state: current_term: 1 leader_uuid: "274337de94a840c09796f28ff1d3b6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } } }
I20260929 14:51:43.225056 20696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274337de94a840c09796f28ff1d3b6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274337de94a840c09796f28ff1d3b6b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35611 } } }
I20260929 14:51:43.225333 20697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:43.225356 20696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:43.226090 20706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:43.226786 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:43.210134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57863 (local address 127.0.0.1:35611)
0929 14:51:43.210644 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:43.210664 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:43.210709 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:51:43.211540 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:43.211562 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:43.212046 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:43.212517 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:43.213394 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:43.214504 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:43.215497 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:43.216044 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:43.216249 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:43.219567 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:43.219574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:43.219594 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:43.219631 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:43.222007 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:43.222804 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:43.222812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:43.222820 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:43.222986 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:43.223428 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:43.223431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:43.223433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:43.224080 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:43.224377 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:43.225048 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 14:51:43.225360 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":168,"threads_started":1}
I20260929 14:51:43.230243 20706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:43.231956 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:43.237836 20706 catalog_manager.cc:1347] Generated new cluster ID: 5ccf4fcdefc64c13946e3dcacf80eb05
I20260929 14:51:43.237923 20706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:43.245052 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:43.234438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37271 (local address 127.0.0.1:35611)
0929 14:51:43.234567 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:43.234572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:43.234860 (+ 288us) server_negotiation.cc:407] Connection header received
0929 14:51:43.234985 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:43.234989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:43.235058 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:43.235180 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:43.236231 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:43.237048 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:43.238470 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:43.238856 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:43.238967 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:43.240949 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:43.240953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:43.240956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:43.240979 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:43.242815 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:43.243399 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:43.243402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:43.243404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:43.243491 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:43.243847 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:43.243849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:43.243851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:43.244062 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:43.244243 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:43.244541 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 14:51:43.244654 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:51:43.255744 20706 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:43.257289 20706 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:43.270795 20706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274337de94a840c09796f28ff1d3b6b4: Generated new TSK 0
I20260929 14:51:43.271672 20706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:43.276611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20622
W20260929 14:51:43.283550 20607 connection.cc:535] client connection to 127.0.0.1:35611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35611): Connection reset by peer (error 104)
2026-09-29T14:51:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:43 dist-test-slave-5hd1 krb5kdc[20721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:43 dist-test-slave-5hd1 krb5kdc[20721](info): set up 2 sockets
Sep 29 14:51:43 dist-test-slave-5hd1 krb5kdc[20721](info): commencing operation
krb5kdc: starting...
W20260929 14:51:44.340606 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:44 dist-test-slave-5hd1 krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:44.488566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:44.607352 20734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:44.607671 20734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:44.607751 20734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:44.611717 20734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:44.611816 20734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:44.611856 20734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:44.611910 20734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:44.611946 20734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:44.617187 20734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:44.618251 20734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:44.619259 20734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:44.629180 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:44.629417 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:44.629179 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:44.630141 20734 server_base.cc:1034] running on GCE node
I20260929 14:51:44.631116 20734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:44.632634 20734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:44.633852 20734 hybrid_clock.cc:648] HybridClock initialized: now 1790693504633821 us; error 54 us; skew 500 ppm
Sep 29 14:51:44 dist-test-slave-5hd1 krb5kdc[20721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:44.637384 20734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:44.638521 20734 webserver.cc:460] Webserver started at http://127.0.0.1:33635/ using document root <none> and password file <none>
I20260929 14:51:44.639228 20734 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:44.639328 20734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:44.639600 20734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:44.642387 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "56cffa6846354e648d7a34ed0baa8f2a"
format_stamp: "Formatted at 2026-09-29 14:51:44 on dist-test-slave-5hd1"
server_key: "14b0289d28c7cbd595dc5c263d061154"
server_key_iv: "7d82557541ee9b2527885450fd823619"
server_key_version: "encryptionkey@0"
I20260929 14:51:44.643079 20734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "56cffa6846354e648d7a34ed0baa8f2a"
format_stamp: "Formatted at 2026-09-29 14:51:44 on dist-test-slave-5hd1"
server_key: "14b0289d28c7cbd595dc5c263d061154"
server_key_iv: "7d82557541ee9b2527885450fd823619"
server_key_version: "encryptionkey@0"
I20260929 14:51:44.647681 20734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 14:51:44.651180 20749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:44.653039 20734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:44.653231 20734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "56cffa6846354e648d7a34ed0baa8f2a"
format_stamp: "Formatted at 2026-09-29 14:51:44 on dist-test-slave-5hd1"
server_key: "14b0289d28c7cbd595dc5c263d061154"
server_key_iv: "7d82557541ee9b2527885450fd823619"
server_key_version: "encryptionkey@0"
I20260929 14:51:44.653393 20734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:44.667807 20734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:44.672366 20734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:44.672640 20734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:44.682265 20734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46121
I20260929 14:51:44.682340 20801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46121 every 8 connection(s)
I20260929 14:51:44.683736 20734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:44.686882 20802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:44.687114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20734
I20260929 14:51:44.687297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:44.687645 392 external_mini_cluster.cc:1428] Setting key 3e9a02b702ede1ffbff6760c172c3b7e
I20260929 14:51:44.693457 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: Bootstrap starting.
I20260929 14:51:44.696411 20802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:44.697202 20802 log.cc:824] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:44.699352 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:44.689380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43921 (local address 127.0.0.1:46121)
0929 14:51:44.690150 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:44.690170 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:44.690216 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:51:44.691048 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:44.691070 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:44.691510 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:44.691868 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:44.692348 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:44.693434 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:44.694693 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:44.695292 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:44.695453 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:44.696661 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:44.696667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:44.696684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:44.697237 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:44.697380 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:44.697433 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:44.697563 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:44.697832 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 14:51:44.698016 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":196,"threads_started":1}
I20260929 14:51:44.699749 20802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: No bootstrap required, opened a new log
I20260929 14:51:44.704242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:44.704210 20802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20260929 14:51:44.704550 20802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:44.704614 20802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cffa6846354e648d7a34ed0baa8f2a, State: Initialized, Role: FOLLOWER
I20260929 14:51:44.705027 20802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20260929 14:51:44.705184 20802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:44.705250 20802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:44.705348 20802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:44.706349 20802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20260929 14:51:44.706719 20802 leader_election.cc:302] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cffa6846354e648d7a34ed0baa8f2a; no voters:
I20260929 14:51:44.707063 20802 leader_election.cc:288] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:44.707233 20811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:44.707667 20811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [term 1 LEADER]: Becoming Leader. State: Replica: 56cffa6846354e648d7a34ed0baa8f2a, State: Running, Role: LEADER
I20260929 14:51:44.708169 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } }
I20260929 14:51:44.708568 20802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:44.711110 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56cffa6846354e648d7a34ed0baa8f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } } }
I20260929 14:51:44.711812 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:44.710668 20813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56cffa6846354e648d7a34ed0baa8f2a. Latest consensus state: current_term: 1 leader_uuid: "56cffa6846354e648d7a34ed0baa8f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cffa6846354e648d7a34ed0baa8f2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46121 } } }
I20260929 14:51:44.711907 20813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a [sys.catalog]: This master's current role is: LEADER
W20260929 14:51:44.713877 20826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:51:44.713971 20826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:51:44.714082 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:44.707993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48435 (local address 127.0.0.1:46121)
0929 14:51:44.708115 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:44.708119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:44.708351 (+ 232us) server_negotiation.cc:407] Connection header received
0929 14:51:44.708533 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:44.708536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:44.708608 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:44.708698 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:44.709157 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:44.710286 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:44.711643 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:44.712014 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:44.712126 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:44.713243 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:44.713247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:44.713249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:44.713261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:44.713290 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:44.713304 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:44.713385 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:44.713812 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 14:51:44.713850 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":41}
I20260929 14:51:44.714469 20828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:44.715368 20828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:44.721881 20828 catalog_manager.cc:1347] Generated new cluster ID: 29dc05178e004b3e953587b1b0b018d2
I20260929 14:51:44.721985 20828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:44.733220 20828 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:44.734241 20828 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:44.744585 20828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56cffa6846354e648d7a34ed0baa8f2a: Generated new TSK 0
I20260929 14:51:44.745455 20828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:44.761849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20734
W20260929 14:51:44.767892 20719 connection.cc:535] client connection to 127.0.0.1:46121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46121): Connection reset by peer (error 104)
2026-09-29T14:51:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:44 dist-test-slave-5hd1 krb5kdc[20833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:44 dist-test-slave-5hd1 krb5kdc[20833](info): set up 2 sockets
Sep 29 14:51:44 dist-test-slave-5hd1 krb5kdc[20833](info): commencing operation
krb5kdc: starting...
W20260929 14:51:45.852648 392 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:45 dist-test-slave-5hd1 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:45.992349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:46.124063 20846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:46.124416 20846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:46.124464 20846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:46.128672 20846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:46.128743 20846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:46.128764 20846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:46.128782 20846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:46.128798 20846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:46.135011 20846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:46.136478 20846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:46.137733 20846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:46.148658 20854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:46.148651 20852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:46.148726 20846 server_base.cc:1034] running on GCE node
W20260929 14:51:46.148648 20851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:46.149672 20846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:46.151019 20846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:46.152262 20846 hybrid_clock.cc:648] HybridClock initialized: now 1790693506152225 us; error 52 us; skew 500 ppm
Sep 29 14:51:46 dist-test-slave-5hd1 krb5kdc[20833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:46.161192 20846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:46.162580 20846 webserver.cc:460] Webserver started at http://127.0.0.1:46291/ using document root <none> and password file <none>
I20260929 14:51:46.163347 20846 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:46.163460 20846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:46.163749 20846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:46.166659 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "150afc29fa6c4502be402c8fe2737eb5"
format_stamp: "Formatted at 2026-09-29 14:51:46 on dist-test-slave-5hd1"
server_key: "c16d6be4b1b5100cc63a16ecba8668b6"
server_key_iv: "58d3e89f0a4b5020ef99a26019a01327"
server_key_version: "encryptionkey@0"
I20260929 14:51:46.167374 20846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "150afc29fa6c4502be402c8fe2737eb5"
format_stamp: "Formatted at 2026-09-29 14:51:46 on dist-test-slave-5hd1"
server_key: "c16d6be4b1b5100cc63a16ecba8668b6"
server_key_iv: "58d3e89f0a4b5020ef99a26019a01327"
server_key_version: "encryptionkey@0"
I20260929 14:51:46.172329 20846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 14:51:46.175511 20861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:46.177240 20846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:46.177603 20846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "150afc29fa6c4502be402c8fe2737eb5"
format_stamp: "Formatted at 2026-09-29 14:51:46 on dist-test-slave-5hd1"
server_key: "c16d6be4b1b5100cc63a16ecba8668b6"
server_key_iv: "58d3e89f0a4b5020ef99a26019a01327"
server_key_version: "encryptionkey@0"
I20260929 14:51:46.177814 20846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:46.183802 20846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:46.188042 20846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:46.188328 20846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:46.197402 20846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45263
I20260929 14:51:46.197435 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45263 every 8 connection(s)
I20260929 14:51:46.198609 20846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:46.200549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20846
I20260929 14:51:46.200706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:46.201050 392 external_mini_cluster.cc:1428] Setting key eb4741ce9b9f3a26ec103cc690ac429c
I20260929 14:51:46.201829 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:46.208143 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:46.202895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50805 (local address 127.0.0.1:45263)
0929 14:51:46.203482 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:46.203503 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:46.203796 (+ 293us) server_negotiation.cc:407] Connection header received
0929 14:51:46.204914 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:46.204937 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:46.205316 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:46.205679 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:46.206627 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:46.206632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:46.206646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:46.206657 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:46.206720 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:46.206742 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:46.206873 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:46.207003 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 14:51:46.207210 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":179,"threads_started":1}
I20260929 14:51:46.209517 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: Bootstrap starting.
I20260929 14:51:46.211776 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:46.211967 20914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:46.212736 20914 log.cc:824] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:46.215086 20914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: No bootstrap required, opened a new log
I20260929 14:51:46.216576 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:46.214677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41659 (local address 127.0.0.1:45263)
0929 14:51:46.214837 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:46.214842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:46.215049 (+ 207us) server_negotiation.cc:407] Connection header received
0929 14:51:46.215164 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:46.215167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:46.215205 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:46.215304 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:46.216121 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:46.216125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:46.216127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:46.216139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:46.216170 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:46.216202 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:46.216300 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:46.216400 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:51:46.216440 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:51:46.218942 20914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260929 14:51:46.219162 20914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:46.219197 20914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 150afc29fa6c4502be402c8fe2737eb5, State: Initialized, Role: FOLLOWER
I20260929 14:51:46.219666 20914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260929 14:51:46.219791 20914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:46.219857 20914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:46.219961 20914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:46.220919 20914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260929 14:51:46.221277 20914 leader_election.cc:302] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 150afc29fa6c4502be402c8fe2737eb5; no voters:
I20260929 14:51:46.221642 20914 leader_election.cc:288] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:46.221849 20924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:46.222464 20924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 150afc29fa6c4502be402c8fe2737eb5, State: Running, Role: LEADER
I20260929 14:51:46.222868 20924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260929 14:51:46.223011 20914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:46.225526 20926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 150afc29fa6c4502be402c8fe2737eb5. Latest consensus state: current_term: 1 leader_uuid: "150afc29fa6c4502be402c8fe2737eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260929 14:51:46.225639 20926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:46.225255 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "150afc29fa6c4502be402c8fe2737eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "150afc29fa6c4502be402c8fe2737eb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260929 14:51:46.225898 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5 [sys.catalog]: This master's current role is: LEADER
W20260929 14:51:46.228116 20938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:51:46.228279 20938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:51:46.228367 20939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:46.229148 20939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:46.236503 20939 catalog_manager.cc:1347] Generated new cluster ID: a35da41dcbaa4b859c1d1055b09fffea
I20260929 14:51:46.236614 20939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:46.260368 20939 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:46.260789 20939 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:46.268450 20939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 150afc29fa6c4502be402c8fe2737eb5: Generated new TSK 0
I20260929 14:51:46.269218 20939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:46.272672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20846
2026-09-29T14:51:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:46 dist-test-slave-5hd1 krb5kdc[20945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:46 dist-test-slave-5hd1 krb5kdc[20945](info): set up 2 sockets
Sep 29 14:51:46 dist-test-slave-5hd1 krb5kdc[20945](info): commencing operation
krb5kdc: starting...
W20260929 14:51:47.328540 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:47.468232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:47.602504 20958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:47.602895 20958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:47.602980 20958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:47.607414 20958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:47.607537 20958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:47.607578 20958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:47.607620 20958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:47.607654 20958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:47.613551 20958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:47.614995 20958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:47.616290 20958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:47.626855 20966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:47.626842 20964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:47.626906 20958 server_base.cc:1034] running on GCE node
W20260929 14:51:47.626843 20963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:47.627974 20958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:47.629426 20958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:47.630681 20958 hybrid_clock.cc:648] HybridClock initialized: now 1790693507630637 us; error 59 us; skew 500 ppm
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[20945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:47.633991 20958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:47.634958 20958 webserver.cc:460] Webserver started at http://127.0.0.1:33569/ using document root <none> and password file <none>
I20260929 14:51:47.635608 20958 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:47.635676 20958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:47.635856 20958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:47.637984 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "685049921bf34a70ace766534387494a"
format_stamp: "Formatted at 2026-09-29 14:51:47 on dist-test-slave-5hd1"
server_key: "466095aa7904e6e505df6c9549c3009f"
server_key_iv: "10744187e1210ded1804f31e06200489"
server_key_version: "encryptionkey@0"
I20260929 14:51:47.638509 20958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "685049921bf34a70ace766534387494a"
format_stamp: "Formatted at 2026-09-29 14:51:47 on dist-test-slave-5hd1"
server_key: "466095aa7904e6e505df6c9549c3009f"
server_key_iv: "10744187e1210ded1804f31e06200489"
server_key_version: "encryptionkey@0"
I20260929 14:51:47.642652 20958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:51:47.645188 20973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:47.646651 20958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:51:47.646799 20958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "685049921bf34a70ace766534387494a"
format_stamp: "Formatted at 2026-09-29 14:51:47 on dist-test-slave-5hd1"
server_key: "466095aa7904e6e505df6c9549c3009f"
server_key_iv: "10744187e1210ded1804f31e06200489"
server_key_version: "encryptionkey@0"
I20260929 14:51:47.646912 20958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:47.665899 20958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:47.669242 20958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:47.669485 20958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:47.677333 20958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34035
I20260929 14:51:47.677440 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34035 every 8 connection(s)
I20260929 14:51:47.678417 20958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:47.681955 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:47.686151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 20958
I20260929 14:51:47.686316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:47.686616 392 external_mini_cluster.cc:1428] Setting key 6c4abf80532ecccf2ff546bf63e92ab5
I20260929 14:51:47.689209 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a: Bootstrap starting.
I20260929 14:51:47.691954 21026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:47.692950 21026 log.cc:824] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:47.695297 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a: No bootstrap required, opened a new log
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[20945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:51:47.699119 21026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260929 14:51:47.699414 21026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:47.699501 21026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685049921bf34a70ace766534387494a, State: Initialized, Role: FOLLOWER
I20260929 14:51:47.700063 21026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260929 14:51:47.700268 21026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:47.700359 21026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:47.700491 21026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:47.701763 21026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260929 14:51:47.702193 21026 leader_election.cc:302] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 685049921bf34a70ace766534387494a; no voters:
I20260929 14:51:47.702607 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:47.688223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51429 (local address 127.0.0.1:34035)
0929 14:51:47.688805 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:47.688823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:47.688866 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:51:47.689763 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:47.689785 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:47.690179 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:47.690529 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:47.691330 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:47.692477 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:47.693396 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:47.693920 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:47.694089 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:47.696774 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:47.696779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:47.696792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:47.696819 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:47.699246 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:47.699793 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:47.699798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:47.699804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:47.699918 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:47.700243 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:47.700249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:47.700251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:47.700830 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:47.701060 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:47.701306 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 14:51:47.701617 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":181,"threads_started":1}
I20260929 14:51:47.702643 21026 leader_election.cc:288] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:47.702724 21031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:47.702925 21031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [term 1 LEADER]: Becoming Leader. State: Replica: 685049921bf34a70ace766534387494a, State: Running, Role: LEADER
I20260929 14:51:47.703382 21031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260929 14:51:47.704316 21026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:47.705449 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "685049921bf34a70ace766534387494a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } } }
I20260929 14:51:47.705929 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:47.705824 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 685049921bf34a70ace766534387494a. Latest consensus state: current_term: 1 leader_uuid: "685049921bf34a70ace766534387494a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "685049921bf34a70ace766534387494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } } }
I20260929 14:51:47.706314 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:47.706954 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:47.707479 21040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:47.711265 21040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:47.717505 21040 catalog_manager.cc:1347] Generated new cluster ID: efca3c3114394b8681176b3fc27affa8
I20260929 14:51:47.717572 21040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:47.725829 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:47.709636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35417 (local address 127.0.0.1:34035)
0929 14:51:47.709751 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:47.709757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:47.716266 (+ 6509us) server_negotiation.cc:407] Connection header received
0929 14:51:47.716500 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:47.716504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:47.716566 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:47.716680 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:47.717629 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:47.718486 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:47.719688 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:47.720076 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:47.720245 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:47.722008 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:47.722012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:47.722015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:47.722039 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:47.723916 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:47.724444 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:47.724448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:47.724450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:47.724532 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:47.724829 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:47.724832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:47.724834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:47.725038 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:47.725157 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:47.725483 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 14:51:47.725653 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:51:47.729075 21040 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:47.730135 21040 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:47.738929 21040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 685049921bf34a70ace766534387494a: Generated new TSK 0
I20260929 14:51:47.739938 21040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:47.749610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 20958
2026-09-29T14:51:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[21057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[21057](info): set up 2 sockets
Sep 29 14:51:47 dist-test-slave-5hd1 krb5kdc[21057](info): commencing operation
krb5kdc: starting...
W20260929 14:51:48.837041 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:48 dist-test-slave-5hd1 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:48.976006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:49.091985 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:49.092355 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:49.092413 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:49.096347 21070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:49.096413 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:49.096432 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:49.096449 21070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:49.096465 21070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:49.101961 21070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:49.103062 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:49.104139 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:51:49.113593 21070 server_base.cc:1034] running on GCE node
W20260929 14:51:49.113485 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:49.113577 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:49.113508 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:49.114423 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:49.115578 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:49.116784 21070 hybrid_clock.cc:648] HybridClock initialized: now 1790693509116772 us; error 54 us; skew 500 ppm
Sep 29 14:51:49 dist-test-slave-5hd1 krb5kdc[21057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:49.119733 21070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:49.120743 21070 webserver.cc:460] Webserver started at http://127.0.0.1:44019/ using document root <none> and password file <none>
I20260929 14:51:49.121323 21070 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:49.121418 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:49.121650 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:49.123677 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "87f30ca3572442649e2d579182c038ac"
format_stamp: "Formatted at 2026-09-29 14:51:49 on dist-test-slave-5hd1"
server_key: "ee55cffe2acad76bbe2c6e46274d7f04"
server_key_iv: "af8be5ed1c6e25f7b97558c2e3dd5c6b"
server_key_version: "encryptionkey@0"
I20260929 14:51:49.124312 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "87f30ca3572442649e2d579182c038ac"
format_stamp: "Formatted at 2026-09-29 14:51:49 on dist-test-slave-5hd1"
server_key: "ee55cffe2acad76bbe2c6e46274d7f04"
server_key_iv: "af8be5ed1c6e25f7b97558c2e3dd5c6b"
server_key_version: "encryptionkey@0"
I20260929 14:51:49.128294 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:51:49.130867 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:49.132449 21070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:49.132606 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "87f30ca3572442649e2d579182c038ac"
format_stamp: "Formatted at 2026-09-29 14:51:49 on dist-test-slave-5hd1"
server_key: "ee55cffe2acad76bbe2c6e46274d7f04"
server_key_iv: "af8be5ed1c6e25f7b97558c2e3dd5c6b"
server_key_version: "encryptionkey@0"
I20260929 14:51:49.132727 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:49.157315 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:49.160470 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:49.160676 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:49.168615 21070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39047
I20260929 14:51:49.168643 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39047 every 8 connection(s)
I20260929 14:51:49.169639 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:49.173234 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:49.174434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21070
I20260929 14:51:49.174602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:49.174913 392 external_mini_cluster.cc:1428] Setting key c47fe5d400e0fd419406446c0d67552e
I20260929 14:51:49.180895 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac: Bootstrap starting.
I20260929 14:51:49.184029 21138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:49.184979 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:49.176756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59023 (local address 127.0.0.1:39047)
0929 14:51:49.177353 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:49.177378 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:49.177418 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:51:49.178274 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:49.178295 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:49.178686 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:49.179052 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:49.179485 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:49.180548 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:49.181475 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:49.181972 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:49.182151 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:49.183130 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:49.183135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:49.183147 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:49.183555 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:49.183606 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:49.183629 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:49.183783 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:49.183928 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 14:51:49.184160 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":198,"threads_started":1}
I20260929 14:51:49.185055 21138 log.cc:824] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:49.187696 21138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac: No bootstrap required, opened a new log
I20260929 14:51:49.188514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:49.193310 21138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } }
I20260929 14:51:49.193608 21138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:49.193683 21138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f30ca3572442649e2d579182c038ac, State: Initialized, Role: FOLLOWER
I20260929 14:51:49.194275 21138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } }
I20260929 14:51:49.194466 21138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:49.194548 21138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:49.194666 21138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:49.195981 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:49.191247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52165 (local address 127.0.0.1:39047)
0929 14:51:49.191435 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:49.191470 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:49.191543 (+ 73us) server_negotiation.cc:407] Connection header received
0929 14:51:49.191657 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:49.191662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:49.191724 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:49.191847 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:49.192085 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:49.192919 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:49.193886 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:49.194236 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:49.194352 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:49.195396 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:49.195401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:49.195405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:49.195419 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:49.195458 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:49.195472 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:49.195563 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:49.195743 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 14:51:49.195825 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:51:49.196223 21138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } }
I20260929 14:51:49.196727 21138 leader_election.cc:302] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f30ca3572442649e2d579182c038ac; no voters:
I20260929 14:51:49.197170 21138 leader_election.cc:288] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:49.197324 21148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:49.197706 21148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [term 1 LEADER]: Becoming Leader. State: Replica: 87f30ca3572442649e2d579182c038ac, State: Running, Role: LEADER
I20260929 14:51:49.198138 21148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } }
I20260929 14:51:49.198971 21138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:49.200165 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f30ca3572442649e2d579182c038ac. Latest consensus state: current_term: 1 leader_uuid: "87f30ca3572442649e2d579182c038ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } } }
I20260929 14:51:49.200202 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f30ca3572442649e2d579182c038ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f30ca3572442649e2d579182c038ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39047 } } }
I20260929 14:51:49.200407 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:49.200769 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:49.200872 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:49.203909 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:49.209373 21157 catalog_manager.cc:1347] Generated new cluster ID: 625aa8fc1d6b4b6280093614d8005fa4
I20260929 14:51:49.209453 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:49.225917 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:49.227667 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:49.238010 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f30ca3572442649e2d579182c038ac: Generated new TSK 0
I20260929 14:51:49.238845 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:49.252359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21070
2026-09-29T14:51:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:49 dist-test-slave-5hd1 krb5kdc[21169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:49 dist-test-slave-5hd1 krb5kdc[21169](info): set up 2 sockets
Sep 29 14:51:49 dist-test-slave-5hd1 krb5kdc[21169](info): commencing operation
krb5kdc: starting...
W20260929 14:51:50.324625 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:50 dist-test-slave-5hd1 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:50.461345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:50.578075 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:50.578365 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:50.578454 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:50.582535 21182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:50.582617 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:50.582664 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:50.582717 21182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:50.582756 21182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:50.588097 21182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:50.589099 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:50.590057 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:50.599103 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:50.599121 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:50.599121 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:50.599527 21182 server_base.cc:1034] running on GCE node
I20260929 14:51:50.600157 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:50.601243 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:50.602440 21182 hybrid_clock.cc:648] HybridClock initialized: now 1790693510602420 us; error 38 us; skew 500 ppm
Sep 29 14:51:50 dist-test-slave-5hd1 krb5kdc[21169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:50.605219 21182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:50.606053 21182 webserver.cc:460] Webserver started at http://127.0.0.1:44711/ using document root <none> and password file <none>
I20260929 14:51:50.606560 21182 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:50.606621 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:50.606778 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:50.608801 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "92e85431a96845159c00ee53f65955a1"
format_stamp: "Formatted at 2026-09-29 14:51:50 on dist-test-slave-5hd1"
server_key: "f49cb059be61acb275e73ec1e04ba47e"
server_key_iv: "5eacccbd08ae308c215af1f4fae61fe4"
server_key_version: "encryptionkey@0"
I20260929 14:51:50.609298 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "92e85431a96845159c00ee53f65955a1"
format_stamp: "Formatted at 2026-09-29 14:51:50 on dist-test-slave-5hd1"
server_key: "f49cb059be61acb275e73ec1e04ba47e"
server_key_iv: "5eacccbd08ae308c215af1f4fae61fe4"
server_key_version: "encryptionkey@0"
I20260929 14:51:50.613004 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:51:50.615504 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:50.617082 21182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:51:50.617242 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "92e85431a96845159c00ee53f65955a1"
format_stamp: "Formatted at 2026-09-29 14:51:50 on dist-test-slave-5hd1"
server_key: "f49cb059be61acb275e73ec1e04ba47e"
server_key_iv: "5eacccbd08ae308c215af1f4fae61fe4"
server_key_version: "encryptionkey@0"
I20260929 14:51:50.617378 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:50.622292 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:50.625515 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:50.625710 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:50.633162 21182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42455
I20260929 14:51:50.633224 21249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42455 every 8 connection(s)
I20260929 14:51:50.634087 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:50.637494 21250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:50.638870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21182
I20260929 14:51:50.638979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:50.639294 392 external_mini_cluster.cc:1428] Setting key deb69a73944b86985fcd14ebca618e54
I20260929 14:51:50.645419 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1: Bootstrap starting.
I20260929 14:51:50.645620 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:50.640877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52247 (local address 127.0.0.1:42455)
0929 14:51:50.641354 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:50.641375 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:50.641444 (+ 69us) server_negotiation.cc:407] Connection header received
0929 14:51:50.642381 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:50.642420 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:50.642823 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:50.643264 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:50.644000 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:50.644006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:50.644021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:50.644033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:50.644104 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:50.644132 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:50.644369 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:50.644454 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 14:51:50.644619 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":119,"threads_started":1}
I20260929 14:51:50.648437 21250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:50.648659 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:50.649312 21250 log.cc:824] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:50.651777 21250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1: No bootstrap required, opened a new log
I20260929 14:51:50.652993 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:50.651059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54233 (local address 127.0.0.1:42455)
0929 14:51:50.651211 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:50.651218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:50.651414 (+ 196us) server_negotiation.cc:407] Connection header received
0929 14:51:50.651594 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:50.651597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:50.651635 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:50.651728 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:50.652531 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:50.652535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:50.652538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:50.652551 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:50.652589 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:50.652602 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:50.652703 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:50.652792 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 14:51:50.652834 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:51:50.655973 21250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } }
I20260929 14:51:50.656322 21250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:50.656418 21250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e85431a96845159c00ee53f65955a1, State: Initialized, Role: FOLLOWER
I20260929 14:51:50.656904 21250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } }
I20260929 14:51:50.657073 21250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:50.657142 21250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:50.657267 21250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:50.658341 21250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } }
I20260929 14:51:50.658767 21250 leader_election.cc:302] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e85431a96845159c00ee53f65955a1; no voters:
I20260929 14:51:50.659157 21250 leader_election.cc:288] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:50.659292 21260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:50.659576 21260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [term 1 LEADER]: Becoming Leader. State: Replica: 92e85431a96845159c00ee53f65955a1, State: Running, Role: LEADER
I20260929 14:51:50.660074 21260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } }
I20260929 14:51:50.660797 21250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:50.661904 21262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e85431a96845159c00ee53f65955a1. Latest consensus state: current_term: 1 leader_uuid: "92e85431a96845159c00ee53f65955a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } } }
I20260929 14:51:50.662047 21261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e85431a96845159c00ee53f65955a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e85431a96845159c00ee53f65955a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42455 } } }
I20260929 14:51:50.662104 21262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:50.662155 21261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:50.662566 21269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:50.665592 21269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:50.671957 21269 catalog_manager.cc:1347] Generated new cluster ID: b15b8fc469724672abde325809458b3f
I20260929 14:51:50.672029 21269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:50.678292 21269 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:50.678597 21269 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:50.683835 21269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e85431a96845159c00ee53f65955a1: Generated new TSK 0
I20260929 14:51:50.684552 21269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:50.689702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21182
2026-09-29T14:51:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:50 dist-test-slave-5hd1 krb5kdc[21281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:50 dist-test-slave-5hd1 krb5kdc[21281](info): set up 2 sockets
Sep 29 14:51:50 dist-test-slave-5hd1 krb5kdc[21281](info): commencing operation
krb5kdc: starting...
W20260929 14:51:51.732585 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:51 dist-test-slave-5hd1 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:51.871587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:51.978561 21294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:51.978852 21294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:51.978920 21294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:51.982975 21294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:51.983067 21294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:51.983106 21294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:51.983165 21294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:51.983201 21294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:51.988571 21294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:51.989614 21294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:51.990628 21294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:51.999567 21300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:51.999583 21299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:51.999640 21294 server_base.cc:1034] running on GCE node
W20260929 14:51:51.999583 21302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:52.000782 21294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:52.002177 21294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:52.003422 21294 hybrid_clock.cc:648] HybridClock initialized: now 1790693512003376 us; error 62 us; skew 500 ppm
Sep 29 14:51:52 dist-test-slave-5hd1 krb5kdc[21281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:52.018743 21294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:52.019920 21294 webserver.cc:460] Webserver started at http://127.0.0.1:32865/ using document root <none> and password file <none>
I20260929 14:51:52.020668 21294 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:52.020740 21294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:52.020947 21294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:52.023414 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "f5245c2ba72f40648e014f9cd00b95fe"
format_stamp: "Formatted at 2026-09-29 14:51:52 on dist-test-slave-5hd1"
server_key: "808c07d7d239b2b318738eebe14084ca"
server_key_iv: "71de4a33cad1afc89368d20de737c1af"
server_key_version: "encryptionkey@0"
I20260929 14:51:52.024042 21294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5245c2ba72f40648e014f9cd00b95fe"
format_stamp: "Formatted at 2026-09-29 14:51:52 on dist-test-slave-5hd1"
server_key: "808c07d7d239b2b318738eebe14084ca"
server_key_iv: "71de4a33cad1afc89368d20de737c1af"
server_key_version: "encryptionkey@0"
I20260929 14:51:52.029040 21294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 14:51:52.032109 21309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:52.033645 21294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:52.033809 21294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "f5245c2ba72f40648e014f9cd00b95fe"
format_stamp: "Formatted at 2026-09-29 14:51:52 on dist-test-slave-5hd1"
server_key: "808c07d7d239b2b318738eebe14084ca"
server_key_iv: "71de4a33cad1afc89368d20de737c1af"
server_key_version: "encryptionkey@0"
I20260929 14:51:52.033954 21294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:52.052909 21294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:52.056600 21294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:52.056841 21294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:52.065806 21294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40281
I20260929 14:51:52.065873 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40281 every 8 connection(s)
I20260929 14:51:52.067086 21294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:52.069628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21294
I20260929 14:51:52.069804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:52.070149 392 external_mini_cluster.cc:1428] Setting key aaa62dfdf81398993259a4c1cb6aaee0
I20260929 14:51:52.070412 21362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:52.077824 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe: Bootstrap starting.
I20260929 14:51:52.080121 21362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe: Neither blocks nor log segments found. Creating new log.
Sep 29 14:51:52 dist-test-slave-5hd1 krb5kdc[21281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:51:52.080902 21362 log.cc:824] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:52.083601 21362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe: No bootstrap required, opened a new log
I20260929 14:51:52.087088 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:52.071969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:40155 (local address 127.0.0.1:40281)
0929 14:51:52.072587 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:52.072605 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:52.072653 (+ 48us) server_negotiation.cc:407] Connection header received
0929 14:51:52.073374 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:52.073396 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:52.073841 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:52.074223 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:52.075021 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:52.076522 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:52.077439 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:52.078111 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:52.078283 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:52.081323 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:52.081329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:52.081346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:52.081383 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:52.083589 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:52.084225 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:52.084230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:52.084236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:52.084376 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:52.084745 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:52.084748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:52.084750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:52.085256 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:52.085467 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:52.085807 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 14:51:52.086105 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":239,"threads_started":1}
I20260929 14:51:52.088099 21362 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260929 14:51:52.088403 21362 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:52.088476 21362 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5245c2ba72f40648e014f9cd00b95fe, State: Initialized, Role: FOLLOWER
I20260929 14:51:52.088994 21362 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260929 14:51:52.089149 21362 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:52.089221 21362 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:52.089323 21362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:52.090445 21362 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260929 14:51:52.090837 21362 leader_election.cc:302] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5245c2ba72f40648e014f9cd00b95fe; no voters:
I20260929 14:51:52.090950 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:52.091204 21362 leader_election.cc:288] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:52.091377 21367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:52.091758 21367 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [term 1 LEADER]: Becoming Leader. State: Replica: f5245c2ba72f40648e014f9cd00b95fe, State: Running, Role: LEADER
I20260929 14:51:52.092257 21367 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260929 14:51:52.092523 21362 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:52.094246 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5245c2ba72f40648e014f9cd00b95fe. Latest consensus state: current_term: 1 leader_uuid: "f5245c2ba72f40648e014f9cd00b95fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260929 14:51:52.094411 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:52.094722 21368 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5245c2ba72f40648e014f9cd00b95fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5245c2ba72f40648e014f9cd00b95fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260929 14:51:52.094818 21368 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:52.095149 21386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:52.098526 21386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:52.104435 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:52.094135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:44269 (local address 127.0.0.1:40281)
0929 14:51:52.094276 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:52.094284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:52.094600 (+ 316us) server_negotiation.cc:407] Connection header received
0929 14:51:52.094807 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:52.094810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:52.094868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:52.094927 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:52.096013 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:52.096764 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:52.098015 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:52.098356 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:52.098471 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:52.100389 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:52.100394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:52.100397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:52.100425 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:52.102449 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:52.103077 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:52.103081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:52.103082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:52.103174 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:52.103511 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:52.103514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:52.103516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:52.103713 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:52.103876 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:52.104136 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 14:51:52.104280 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:51:52.109972 21386 catalog_manager.cc:1347] Generated new cluster ID: 765926bbdc1d414f888b0a82d264c576
I20260929 14:51:52.110071 21386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:52.119519 21386 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:52.120594 21386 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:52.127653 21386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5245c2ba72f40648e014f9cd00b95fe: Generated new TSK 0
I20260929 14:51:52.128533 21386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:52.138370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21294
W20260929 14:51:52.144481 21279 connection.cc:535] client connection to 127.0.0.1:40281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40281): Connection reset by peer (error 104)
2026-09-29T14:51:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:52 dist-test-slave-5hd1 krb5kdc[21393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:52 dist-test-slave-5hd1 krb5kdc[21393](info): set up 2 sockets
Sep 29 14:51:52 dist-test-slave-5hd1 krb5kdc[21393](info): commencing operation
krb5kdc: starting...
W20260929 14:51:53.192619 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:53 dist-test-slave-5hd1 krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:53.324538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:53.434484 21406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:53.434769 21406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:53.434811 21406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:53.439208 21406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:53.439294 21406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:53.439316 21406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:53.439333 21406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:53.439349 21406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:53.444936 21406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:53.446097 21406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:53.447124 21406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:51:53.456766 21406 server_base.cc:1034] running on GCE node
W20260929 14:51:53.456676 21412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:53.456730 21411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:53.456676 21414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:53.457649 21406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:53.459000 21406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:53.460243 21406 hybrid_clock.cc:648] HybridClock initialized: now 1790693513460202 us; error 56 us; skew 500 ppm
Sep 29 14:51:53 dist-test-slave-5hd1 krb5kdc[21393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:53.463280 21406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:53.464586 21406 webserver.cc:460] Webserver started at http://127.0.0.1:40077/ using document root <none> and password file <none>
I20260929 14:51:53.465247 21406 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:53.465425 21406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:53.465852 21406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:53.468654 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "dda5d1232bc845efac48ecb631fa9364"
format_stamp: "Formatted at 2026-09-29 14:51:53 on dist-test-slave-5hd1"
server_key: "8f25bde4b5ba2d761e3f6534098fb3aa"
server_key_iv: "4e0dde4c12b2e5cecbc7455114ac661f"
server_key_version: "encryptionkey@0"
I20260929 14:51:53.469327 21406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "dda5d1232bc845efac48ecb631fa9364"
format_stamp: "Formatted at 2026-09-29 14:51:53 on dist-test-slave-5hd1"
server_key: "8f25bde4b5ba2d761e3f6534098fb3aa"
server_key_iv: "4e0dde4c12b2e5cecbc7455114ac661f"
server_key_version: "encryptionkey@0"
I20260929 14:51:53.474532 21406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:51:53.477891 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:53.479584 21406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:53.479735 21406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "dda5d1232bc845efac48ecb631fa9364"
format_stamp: "Formatted at 2026-09-29 14:51:53 on dist-test-slave-5hd1"
server_key: "8f25bde4b5ba2d761e3f6534098fb3aa"
server_key_iv: "4e0dde4c12b2e5cecbc7455114ac661f"
server_key_version: "encryptionkey@0"
I20260929 14:51:53.479876 21406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:53.494247 21406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:53.498234 21406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:53.498440 21406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:53.507617 21406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37071
I20260929 14:51:53.507644 21473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37071 every 8 connection(s)
I20260929 14:51:53.509060 21406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:53.511247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21406
I20260929 14:51:53.511449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:53.511865 392 external_mini_cluster.cc:1428] Setting key a50f97ce9f90075c34154f1e23a59980
I20260929 14:51:53.512400 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:53.519390 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364: Bootstrap starting.
I20260929 14:51:53.521843 21474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:53.522599 21474 log.cc:824] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:53.524037 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:53.513800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:52897 (local address 127.0.0.1:37071)
0929 14:51:53.514355 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:53.514377 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:53.514433 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:51:53.515383 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:53.515407 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:53.515855 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:53.516295 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:53.516798 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:53.517955 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:53.518912 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:53.519455 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:53.519644 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:53.521038 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:53.521043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:53.521061 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:53.522111 (+ 1050us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:53.522191 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:53.522230 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:53.522500 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:53.522692 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 14:51:53.522883 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":179,"threads_started":1}
I20260929 14:51:53.525307 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364: No bootstrap required, opened a new log
I20260929 14:51:53.528654 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:53.529481 21474 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260929 14:51:53.529759 21474 raft_consensus.cc:374] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:53.529796 21474 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda5d1232bc845efac48ecb631fa9364, State: Initialized, Role: FOLLOWER
I20260929 14:51:53.530478 21474 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260929 14:51:53.530648 21474 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:53.530756 21474 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:53.530851 21474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:53.532493 21474 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260929 14:51:53.532948 21474 leader_election.cc:302] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda5d1232bc845efac48ecb631fa9364; no voters:
I20260929 14:51:53.533351 21474 leader_election.cc:288] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:53.533443 21484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:53.533627 21484 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [term 1 LEADER]: Becoming Leader. State: Replica: dda5d1232bc845efac48ecb631fa9364, State: Running, Role: LEADER
I20260929 14:51:53.534020 21484 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } }
I20260929 14:51:53.534750 21474 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:53.535326 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda5d1232bc845efac48ecb631fa9364. Latest consensus state: current_term: 1 leader_uuid: "dda5d1232bc845efac48ecb631fa9364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260929 14:51:53.535627 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:53.535902 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dda5d1232bc845efac48ecb631fa9364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda5d1232bc845efac48ecb631fa9364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37071 } } }
I20260929 14:51:53.536101 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:53.536495 21489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:53.537854 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:53.531751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:57901 (local address 127.0.0.1:37071)
0929 14:51:53.531951 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:53.531957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:53.532236 (+ 279us) server_negotiation.cc:407] Connection header received
0929 14:51:53.532436 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:53.532440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:53.532505 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:53.532599 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:53.532907 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:53.533660 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:53.534784 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:53.535118 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:53.535208 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:53.536938 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:53.536943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:53.536946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:53.536966 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:53.537008 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:53.537025 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:53.537114 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:53.537349 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 14:51:53.537413 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":78}
I20260929 14:51:53.540789 21489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:53.547633 21489 catalog_manager.cc:1347] Generated new cluster ID: b14ea7fb31944d5fa48102f81b8284d0
I20260929 14:51:53.547734 21489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:53.556603 21489 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:53.557623 21489 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:53.567030 21489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dda5d1232bc845efac48ecb631fa9364: Generated new TSK 0
I20260929 14:51:53.567981 21489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:53.576053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21406
W20260929 14:51:53.582326 21391 connection.cc:535] client connection to 127.0.0.1:37071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37071): Connection reset by peer (error 104)
2026-09-29T14:51:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:53 dist-test-slave-5hd1 krb5kdc[21505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:53 dist-test-slave-5hd1 krb5kdc[21505](info): set up 2 sockets
Sep 29 14:51:53 dist-test-slave-5hd1 krb5kdc[21505](info): commencing operation
krb5kdc: starting...
W20260929 14:51:54.628618 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:54 dist-test-slave-5hd1 krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:54.767208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:54.893513 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:54.893836 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:54.893883 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:54.898041 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:54.898116 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:54.898136 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:54.898154 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:54.898170 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:54.903993 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:54.905390 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:54.906713 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:54.916944 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:54.916929 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:54.917140 21518 server_base.cc:1034] running on GCE node
W20260929 14:51:54.916919 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:54.918030 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:54.919207 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:54.920404 21518 hybrid_clock.cc:648] HybridClock initialized: now 1790693514920391 us; error 34 us; skew 500 ppm
Sep 29 14:51:54 dist-test-slave-5hd1 krb5kdc[21505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:54.923429 21518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:54.924463 21518 webserver.cc:460] Webserver started at http://127.0.0.1:42439/ using document root <none> and password file <none>
I20260929 14:51:54.925068 21518 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:54.925165 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:54.925395 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:54.927444 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "d98305314bd84e1b89183e9a02d985f8"
format_stamp: "Formatted at 2026-09-29 14:51:54 on dist-test-slave-5hd1"
server_key: "4cdb2a596b839a5e8f2f665d845cff48"
server_key_iv: "adf44a1615fa31bb3fcff07b6ab4aed7"
server_key_version: "encryptionkey@0"
I20260929 14:51:54.928009 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "d98305314bd84e1b89183e9a02d985f8"
format_stamp: "Formatted at 2026-09-29 14:51:54 on dist-test-slave-5hd1"
server_key: "4cdb2a596b839a5e8f2f665d845cff48"
server_key_iv: "adf44a1615fa31bb3fcff07b6ab4aed7"
server_key_version: "encryptionkey@0"
I20260929 14:51:54.932514 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:51:54.935189 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:54.936712 21518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:54.936856 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "d98305314bd84e1b89183e9a02d985f8"
format_stamp: "Formatted at 2026-09-29 14:51:54 on dist-test-slave-5hd1"
server_key: "4cdb2a596b839a5e8f2f665d845cff48"
server_key_iv: "adf44a1615fa31bb3fcff07b6ab4aed7"
server_key_version: "encryptionkey@0"
I20260929 14:51:54.936969 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:54.941614 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:54.944821 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:54.945011 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:54.952433 21518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45197
I20260929 14:51:54.952565 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45197 every 8 connection(s)
I20260929 14:51:54.953506 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:54.954505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21518
I20260929 14:51:54.954675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:54.954974 392 external_mini_cluster.cc:1428] Setting key 66f1007341a9b074a5054c77ae76d562
I20260929 14:51:54.957544 21586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:54.961560 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:54.956570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:47207 (local address 127.0.0.1:45197)
0929 14:51:54.957048 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:54.957071 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:54.957126 (+ 55us) server_negotiation.cc:407] Connection header received
0929 14:51:54.958155 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:54.958182 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:54.958580 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:54.959013 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:54.959915 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:54.959922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:54.959937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:54.959949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:54.960022 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:54.960049 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:54.960224 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:54.960334 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 14:51:54.960511 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":120,"threads_started":1}
I20260929 14:51:54.964978 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:54.966020 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8: Bootstrap starting.
I20260929 14:51:54.968844 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:54.967235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:34829 (local address 127.0.0.1:45197)
0929 14:51:54.967404 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:54.967409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:54.967505 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:51:54.967609 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:54.967613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:54.967665 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:54.967763 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:54.968441 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:54.968444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:54.968448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:54.968459 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:54.968491 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:54.968504 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:54.968602 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:54.968662 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 14:51:54.968707 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:51:54.969267 21586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:54.970136 21586 log.cc:824] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:54.972517 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8: No bootstrap required, opened a new log
I20260929 14:51:54.976769 21586 raft_consensus.cc:348] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 14:51:54.977072 21586 raft_consensus.cc:374] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:54.977160 21586 raft_consensus.cc:729] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98305314bd84e1b89183e9a02d985f8, State: Initialized, Role: FOLLOWER
I20260929 14:51:54.977730 21586 consensus_queue.cc:260] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 14:51:54.977900 21586 raft_consensus.cc:388] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:54.977991 21586 raft_consensus.cc:482] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:54.978116 21586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:54.979271 21586 raft_consensus.cc:504] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 14:51:54.979734 21586 leader_election.cc:302] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d98305314bd84e1b89183e9a02d985f8; no voters:
I20260929 14:51:54.980152 21586 leader_election.cc:288] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:54.980453 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:54.980818 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [term 1 LEADER]: Becoming Leader. State: Replica: d98305314bd84e1b89183e9a02d985f8, State: Running, Role: LEADER
I20260929 14:51:54.981336 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } }
I20260929 14:51:54.981906 21586 sys_catalog.cc:564] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:54.983553 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d98305314bd84e1b89183e9a02d985f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } } }
I20260929 14:51:54.983716 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:54.984162 21605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:54.984123 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d98305314bd84e1b89183e9a02d985f8. Latest consensus state: current_term: 1 leader_uuid: "d98305314bd84e1b89183e9a02d985f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98305314bd84e1b89183e9a02d985f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45197 } } }
I20260929 14:51:54.984303 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:54.988811 21605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:54.995878 21605 catalog_manager.cc:1347] Generated new cluster ID: 6500ebbb6fe74e70adf2d0737d2b1a25
I20260929 14:51:54.995986 21605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:55.028004 21605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:55.028411 21605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:55.034618 21605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d98305314bd84e1b89183e9a02d985f8: Generated new TSK 0
I20260929 14:51:55.035363 21605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:55.045821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21518
2026-09-29T14:51:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:55 dist-test-slave-5hd1 krb5kdc[21617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:55 dist-test-slave-5hd1 krb5kdc[21617](info): set up 2 sockets
Sep 29 14:51:55 dist-test-slave-5hd1 krb5kdc[21617](info): commencing operation
krb5kdc: starting...
W20260929 14:51:56.096590 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:56.239845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:56.368691 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:56.369030 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:56.369077 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:56.373314 21630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:56.373394 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:56.373414 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:56.373433 21630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:56.373449 21630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:56.379151 21630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:56.380560 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:56.381851 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:56.392294 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:56.392272 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:56.392346 21630 server_base.cc:1034] running on GCE node
W20260929 14:51:56.392268 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:56.393481 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:56.394836 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:56.396050 21630 hybrid_clock.cc:648] HybridClock initialized: now 1790693516396032 us; error 35 us; skew 500 ppm
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:56.399219 21630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:56.400141 21630 webserver.cc:460] Webserver started at http://127.0.0.1:41549/ using document root <none> and password file <none>
I20260929 14:51:56.400807 21630 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:56.400882 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:56.401057 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:56.403213 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "8a921892a2e145bd924feac1432639ea"
format_stamp: "Formatted at 2026-09-29 14:51:56 on dist-test-slave-5hd1"
server_key: "c14020ad6cb1b0a3ba1b07bc5e8d495c"
server_key_iv: "8c3ae5393e1fe7e82d68eb863b38ec28"
server_key_version: "encryptionkey@0"
I20260929 14:51:56.403761 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a921892a2e145bd924feac1432639ea"
format_stamp: "Formatted at 2026-09-29 14:51:56 on dist-test-slave-5hd1"
server_key: "c14020ad6cb1b0a3ba1b07bc5e8d495c"
server_key_iv: "8c3ae5393e1fe7e82d68eb863b38ec28"
server_key_version: "encryptionkey@0"
I20260929 14:51:56.407783 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:51:56.410557 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:56.411994 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 14:51:56.412135 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "8a921892a2e145bd924feac1432639ea"
format_stamp: "Formatted at 2026-09-29 14:51:56 on dist-test-slave-5hd1"
server_key: "c14020ad6cb1b0a3ba1b07bc5e8d495c"
server_key_iv: "8c3ae5393e1fe7e82d68eb863b38ec28"
server_key_version: "encryptionkey@0"
I20260929 14:51:56.412308 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:56.448307 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:56.451603 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:56.451761 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:56.459622 21630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39555
I20260929 14:51:56.459666 21697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39555 every 8 connection(s)
I20260929 14:51:56.460705 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:56.464241 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:56.468156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21630
I20260929 14:51:56.468447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:56.468783 392 external_mini_cluster.cc:1428] Setting key eb6a0a87469b9a8990312d9674a76376
I20260929 14:51:56.471791 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: Bootstrap starting.
I20260929 14:51:56.475067 21698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:56.476248 21698 log.cc:824] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:56.479213 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: No bootstrap required, opened a new log
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:51:56.482883 21698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260929 14:51:56.483199 21698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:56.483264 21698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a921892a2e145bd924feac1432639ea, State: Initialized, Role: FOLLOWER
I20260929 14:51:56.483851 21698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260929 14:51:56.484025 21698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:56.484098 21698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:56.484251 21698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:56.485792 21698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260929 14:51:56.486223 21698 leader_election.cc:302] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a921892a2e145bd924feac1432639ea; no voters:
I20260929 14:51:56.486622 21698 leader_election.cc:288] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:56.486805 21703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:56.487140 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:56.470671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:47395 (local address 127.0.0.1:39555)
0929 14:51:56.471198 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:56.471218 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:56.471266 (+ 48us) server_negotiation.cc:407] Connection header received
0929 14:51:56.472136 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:56.472156 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:56.472620 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:56.472932 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:56.474146 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:56.475228 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:56.476559 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:56.477250 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:56.477404 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:56.480643 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:56.480649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:56.480664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:56.480693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:56.483039 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:56.483734 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:56.483741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:56.483747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:56.483878 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:56.484329 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:56.484335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:56.484337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:56.484975 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:56.485254 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:56.485647 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 14:51:56.485954 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":182,"threads_started":1}
I20260929 14:51:56.488065 21698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:56.487136 21703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [term 1 LEADER]: Becoming Leader. State: Replica: 8a921892a2e145bd924feac1432639ea, State: Running, Role: LEADER
I20260929 14:51:56.488847 21703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260929 14:51:56.491325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:56.490839 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a921892a2e145bd924feac1432639ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } } }
I20260929 14:51:56.491601 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:56.491318 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a921892a2e145bd924feac1432639ea. Latest consensus state: current_term: 1 leader_uuid: "8a921892a2e145bd924feac1432639ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a921892a2e145bd924feac1432639ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } } }
I20260929 14:51:56.491798 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea [sys.catalog]: This master's current role is: LEADER
W20260929 14:51:56.494340 21717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:51:56.494439 21717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:51:56.494580 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:56.495684 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:56.502702 21718 catalog_manager.cc:1347] Generated new cluster ID: 2c2100193c0f4e7fa1ec278767f201b2
I20260929 14:51:56.502830 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:56.502996 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:56.493938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:45699 (local address 127.0.0.1:39555)
0929 14:51:56.494093 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:56.494099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:56.494367 (+ 268us) server_negotiation.cc:407] Connection header received
0929 14:51:56.494564 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:56.494568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:56.494621 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:56.494679 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:56.495521 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:56.496112 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:56.497103 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:56.497382 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:56.497503 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:56.499126 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:56.499130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:56.499133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:51:56.499161 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:56.501064 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:56.501561 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:56.501565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:56.501567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:56.501654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:51:56.502003 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:51:56.502006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:51:56.502008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:51:56.502211 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:56.502377 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:56.502694 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 14:51:56.502810 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:51:56.511372 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:56.512682 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:56.518366 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a921892a2e145bd924feac1432639ea: Generated new TSK 0
I20260929 14:51:56.519345 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:56.530776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21630
W20260929 14:51:56.536689 21615 connection.cc:535] client connection to 127.0.0.1:39555 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39555): Connection reset by peer (error 104)
2026-09-29T14:51:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21729](info): set up 2 sockets
Sep 29 14:51:56 dist-test-slave-5hd1 krb5kdc[21729](info): commencing operation
krb5kdc: starting...
W20260929 14:51:57.576584 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:57 dist-test-slave-5hd1 krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:57.716699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:57.833657 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:57.833976 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:57.834059 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:57.838474 21742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:57.838594 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:57.838634 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:57.838690 21742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:57.838728 21742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:57.844151 21742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:57.845381 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:57.846490 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:57.855641 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:57.855749 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:57.855729 21742 server_base.cc:1034] running on GCE node
W20260929 14:51:57.856024 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:57.856737 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:57.858074 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:57.859251 21742 hybrid_clock.cc:648] HybridClock initialized: now 1790693517859228 us; error 38 us; skew 500 ppm
Sep 29 14:51:57 dist-test-slave-5hd1 krb5kdc[21729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:57.868870 21742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:57.870136 21742 webserver.cc:460] Webserver started at http://127.0.0.1:34319/ using document root <none> and password file <none>
I20260929 14:51:57.870865 21742 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:57.870967 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:57.871244 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:57.873847 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "ce1cabe824cd4d2aaae65d047562963c"
format_stamp: "Formatted at 2026-09-29 14:51:57 on dist-test-slave-5hd1"
server_key: "1da9348743229cd6363939bb9e19a6ea"
server_key_iv: "14e4eeadd056ec190910225cad2d3ed2"
server_key_version: "encryptionkey@0"
I20260929 14:51:57.874521 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce1cabe824cd4d2aaae65d047562963c"
format_stamp: "Formatted at 2026-09-29 14:51:57 on dist-test-slave-5hd1"
server_key: "1da9348743229cd6363939bb9e19a6ea"
server_key_iv: "14e4eeadd056ec190910225cad2d3ed2"
server_key_version: "encryptionkey@0"
I20260929 14:51:57.879110 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:51:57.882001 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:57.883566 21742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:51:57.883728 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "ce1cabe824cd4d2aaae65d047562963c"
format_stamp: "Formatted at 2026-09-29 14:51:57 on dist-test-slave-5hd1"
server_key: "1da9348743229cd6363939bb9e19a6ea"
server_key_iv: "14e4eeadd056ec190910225cad2d3ed2"
server_key_version: "encryptionkey@0"
I20260929 14:51:57.883858 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:57.917258 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:57.920831 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:57.921064 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:57.929452 21742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43905
I20260929 14:51:57.929486 21809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43905 every 8 connection(s)
I20260929 14:51:57.930598 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:57.933712 21810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:57.935030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21742
I20260929 14:51:57.935163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:57.935544 392 external_mini_cluster.cc:1428] Setting key 37831ead6908b6fc1c131391b4338cc0
I20260929 14:51:57.941309 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c: Bootstrap starting.
I20260929 14:51:57.944557 21810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:57.945401 21810 log.cc:824] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:57.947331 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:57.937443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:51549 (local address 127.0.0.1:43905)
0929 14:51:57.938026 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:57.938050 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:57.938103 (+ 53us) server_negotiation.cc:407] Connection header received
0929 14:51:57.939006 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:57.939047 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:57.939519 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:57.939938 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:57.940452 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:57.941918 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:57.943051 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:57.943956 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:57.944163 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:57.945350 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:57.945357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:57.945374 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:57.945897 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:57.945968 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:57.945992 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:57.946161 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:57.946402 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 14:51:57.946550 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":143,"threads_started":1}
I20260929 14:51:57.948388 21810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c: No bootstrap required, opened a new log
I20260929 14:51:57.951552 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:57.953697 21810 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } }
I20260929 14:51:57.954130 21810 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:57.954196 21810 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1cabe824cd4d2aaae65d047562963c, State: Initialized, Role: FOLLOWER
I20260929 14:51:57.954813 21810 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } }
I20260929 14:51:57.955022 21810 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:57.955122 21810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:57.955271 21810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:57.956822 21810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } }
I20260929 14:51:57.957273 21810 leader_election.cc:302] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce1cabe824cd4d2aaae65d047562963c; no voters:
I20260929 14:51:57.957722 21810 leader_election.cc:288] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:57.957876 21820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:57.958189 21820 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [term 1 LEADER]: Becoming Leader. State: Replica: ce1cabe824cd4d2aaae65d047562963c, State: Running, Role: LEADER
I20260929 14:51:57.958648 21820 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } }
I20260929 14:51:57.959280 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:57.953847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:44873 (local address 127.0.0.1:43905)
0929 14:51:57.953982 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:57.953985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:57.954338 (+ 353us) server_negotiation.cc:407] Connection header received
0929 14:51:57.954389 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:57.954392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:57.954439 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:57.954577 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:57.954932 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:57.955694 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:57.956988 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:51:57.957279 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:51:57.957369 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:51:57.958626 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:57.958631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:57.958634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:57.958650 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:57.958685 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:57.958699 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:57.958795 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:57.959085 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 14:51:57.959131 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:51:57.959893 21810 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:57.960227 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce1cabe824cd4d2aaae65d047562963c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } } }
I20260929 14:51:57.960361 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:57.960767 21824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:57.960995 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1cabe824cd4d2aaae65d047562963c. Latest consensus state: current_term: 1 leader_uuid: "ce1cabe824cd4d2aaae65d047562963c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1cabe824cd4d2aaae65d047562963c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43905 } } }
I20260929 14:51:57.961134 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:57.963965 21824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:57.970530 21824 catalog_manager.cc:1347] Generated new cluster ID: d821511c1eee42f79028f1893c0d3d7f
I20260929 14:51:57.970607 21824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:57.980743 21824 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:57.981753 21824 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:57.992002 21824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce1cabe824cd4d2aaae65d047562963c: Generated new TSK 0
I20260929 14:51:57.992877 21824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:57.997673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21742
W20260929 14:51:58.003526 21727 connection.cc:535] client connection to 127.0.0.1:43905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43905): Connection reset by peer (error 104)
2026-09-29T14:51:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:58 dist-test-slave-5hd1 krb5kdc[21841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:58 dist-test-slave-5hd1 krb5kdc[21841](info): set up 2 sockets
Sep 29 14:51:58 dist-test-slave-5hd1 krb5kdc[21841](info): commencing operation
krb5kdc: starting...
W20260929 14:51:59.076588 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:51:59 dist-test-slave-5hd1 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:51:59.215914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:51:59.341754 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:51:59.342170 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:51:59.342254 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:51:59.346616 21854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:51:59.346742 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:51:59.346778 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:51:59.346809 21854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:51:59.346863 21854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:51:59.353018 21854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:51:59.354627 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:51:59.356035 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:51:59.366156 21860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:51:59.366174 21862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:59.366437 21854 server_base.cc:1034] running on GCE node
W20260929 14:51:59.366174 21859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:51:59.367311 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:51:59.368757 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:51:59.369979 21854 hybrid_clock.cc:648] HybridClock initialized: now 1790693519369953 us; error 47 us; skew 500 ppm
Sep 29 14:51:59 dist-test-slave-5hd1 krb5kdc[21841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:51:59.373348 21854 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:51:59.374387 21854 webserver.cc:460] Webserver started at http://127.0.0.1:37975/ using document root <none> and password file <none>
I20260929 14:51:59.374974 21854 fs_manager.cc:359] Metadata directory not provided
I20260929 14:51:59.375036 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:51:59.375222 21854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:51:59.377367 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "f215f2e58211495295f0f0d3f4451049"
format_stamp: "Formatted at 2026-09-29 14:51:59 on dist-test-slave-5hd1"
server_key: "a822c4fb0cce3f4f9fdaca8add64dd1f"
server_key_iv: "960c3f25ee4710d9679f5ac77c4b0264"
server_key_version: "encryptionkey@0"
I20260929 14:51:59.377893 21854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "f215f2e58211495295f0f0d3f4451049"
format_stamp: "Formatted at 2026-09-29 14:51:59 on dist-test-slave-5hd1"
server_key: "a822c4fb0cce3f4f9fdaca8add64dd1f"
server_key_iv: "960c3f25ee4710d9679f5ac77c4b0264"
server_key_version: "encryptionkey@0"
I20260929 14:51:59.381860 21854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:51:59.384660 21869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:51:59.386070 21854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:51:59.386246 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "f215f2e58211495295f0f0d3f4451049"
format_stamp: "Formatted at 2026-09-29 14:51:59 on dist-test-slave-5hd1"
server_key: "a822c4fb0cce3f4f9fdaca8add64dd1f"
server_key_iv: "960c3f25ee4710d9679f5ac77c4b0264"
server_key_version: "encryptionkey@0"
I20260929 14:51:59.386510 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:51:59.391281 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:51:59.394707 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:51:59.394909 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:51:59.402608 21854 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33957
I20260929 14:51:59.402758 21921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33957 every 8 connection(s)
I20260929 14:51:59.403626 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:51:59.407004 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:51:59.413388 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049: Bootstrap starting.
I20260929 14:51:59.413897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21854
I20260929 14:51:59.414060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:51:59.414386 392 external_mini_cluster.cc:1428] Setting key 8208eed126e41565b5f0e0a0f74ef735
I20260929 14:51:59.416481 21922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049: Neither blocks nor log segments found. Creating new log.
I20260929 14:51:59.417315 21922 log.cc:824] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049: Log is configured to *not* fsync() on all Append() calls
I20260929 14:51:59.419582 21922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049: No bootstrap required, opened a new log
I20260929 14:51:59.420475 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:59.415794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:54609 (local address 127.0.0.1:33957)
0929 14:51:59.416447 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:59.416465 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:59.416511 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:51:59.417460 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:59.417496 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:59.417836 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:59.418197 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:59.419057 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:59.419066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:59.419082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:59.419093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:59.419149 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:59.419169 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:59.419293 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:59.419395 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 14:51:59.419569 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":190,"threads_started":1}
I20260929 14:51:59.423851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:51:59.424418 21922 raft_consensus.cc:348] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260929 14:51:59.424669 21922 raft_consensus.cc:374] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:51:59.424736 21922 raft_consensus.cc:729] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215f2e58211495295f0f0d3f4451049, State: Initialized, Role: FOLLOWER
I20260929 14:51:59.425252 21922 consensus_queue.cc:260] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260929 14:51:59.425436 21922 raft_consensus.cc:388] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:51:59.425561 21922 raft_consensus.cc:482] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:51:59.425693 21922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:51:59.427116 21922 raft_consensus.cc:504] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260929 14:51:59.427546 21922 leader_election.cc:302] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f215f2e58211495295f0f0d3f4451049; no voters:
I20260929 14:51:59.427961 21922 leader_election.cc:288] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:51:59.428146 21932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:51:59.428208 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:51:59.426284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:56651 (local address 127.0.0.1:33957)
0929 14:51:59.426409 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:51:59.426414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:51:59.426539 (+ 125us) server_negotiation.cc:407] Connection header received
0929 14:51:59.426629 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:51:59.426632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:51:59.426682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:51:59.426791 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:51:59.427673 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:51:59.427677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:51:59.427680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:51:59.427691 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:51:59.427723 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:51:59.427736 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:51:59.427871 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:51:59.427996 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 14:51:59.428056 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:51:59.428712 21932 raft_consensus.cc:686] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [term 1 LEADER]: Becoming Leader. State: Replica: f215f2e58211495295f0f0d3f4451049, State: Running, Role: LEADER
I20260929 14:51:59.429172 21932 consensus_queue.cc:237] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260929 14:51:59.429605 21922 sys_catalog.cc:564] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:51:59.431188 21934 sys_catalog.cc:455] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f215f2e58211495295f0f0d3f4451049. Latest consensus state: current_term: 1 leader_uuid: "f215f2e58211495295f0f0d3f4451049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260929 14:51:59.431353 21934 sys_catalog.cc:458] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:59.431758 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f215f2e58211495295f0f0d3f4451049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f215f2e58211495295f0f0d3f4451049" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260929 14:51:59.431828 21943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:51:59.431859 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049 [sys.catalog]: This master's current role is: LEADER
I20260929 14:51:59.434986 21943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:51:59.440721 21943 catalog_manager.cc:1347] Generated new cluster ID: 271ab6eabfeb4f53b69a50f23599fc91
I20260929 14:51:59.440793 21943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:51:59.456622 21943 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:51:59.457074 21943 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:51:59.470608 21943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f215f2e58211495295f0f0d3f4451049: Generated new TSK 0
I20260929 14:51:59.471706 21943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:51:59.476027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21854
2026-09-29T14:51:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:51:59 dist-test-slave-5hd1 krb5kdc[21953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:51:59 dist-test-slave-5hd1 krb5kdc[21953](info): set up 2 sockets
Sep 29 14:51:59 dist-test-slave-5hd1 krb5kdc[21953](info): commencing operation
krb5kdc: starting...
W20260929 14:52:00.540994 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:00 dist-test-slave-5hd1 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:00.679368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:00.799773 21966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:00.800200 21966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:00.800341 21966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:00.804955 21966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:00.805081 21966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:00.805128 21966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:00.805202 21966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:00.805245 21966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:00.811273 21966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:00.812557 21966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:00.813705 21966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:00.823810 21971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:00.823753 21974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:00.824095 21966 server_base.cc:1034] running on GCE node
W20260929 14:52:00.823755 21972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:00.824816 21966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:00.826002 21966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:00.827200 21966 hybrid_clock.cc:648] HybridClock initialized: now 1790693520827184 us; error 43 us; skew 500 ppm
Sep 29 14:52:00 dist-test-slave-5hd1 krb5kdc[21953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:00.830298 21966 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:00.831319 21966 webserver.cc:460] Webserver started at http://127.0.98.62:46471/ using document root <none> and password file <none>
I20260929 14:52:00.831914 21966 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:00.832015 21966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:00.832309 21966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:00.834373 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "81d5bf8848eb4c038899ef8c763309c4"
format_stamp: "Formatted at 2026-09-29 14:52:00 on dist-test-slave-5hd1"
server_key: "00266b36b69c13b09af740c560ef4928"
server_key_iv: "a16a07603dd59f7b6e908c7607e3f8a6"
server_key_version: "encryptionkey@0"
I20260929 14:52:00.834976 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "81d5bf8848eb4c038899ef8c763309c4"
format_stamp: "Formatted at 2026-09-29 14:52:00 on dist-test-slave-5hd1"
server_key: "00266b36b69c13b09af740c560ef4928"
server_key_iv: "a16a07603dd59f7b6e908c7607e3f8a6"
server_key_version: "encryptionkey@0"
I20260929 14:52:00.838969 21966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:52:00.841879 21981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:00.843281 21966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:52:00.843439 21966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "81d5bf8848eb4c038899ef8c763309c4"
format_stamp: "Formatted at 2026-09-29 14:52:00 on dist-test-slave-5hd1"
server_key: "00266b36b69c13b09af740c560ef4928"
server_key_iv: "a16a07603dd59f7b6e908c7607e3f8a6"
server_key_version: "encryptionkey@0"
I20260929 14:52:00.843555 21966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:00.866875 21966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:00.870097 21966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:00.870309 21966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:00.877923 21966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46051
I20260929 14:52:00.877952 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46051 every 8 connection(s)
I20260929 14:52:00.878849 21966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:00.882205 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:00.887439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 21966
I20260929 14:52:00.887606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:00.887929 392 external_mini_cluster.cc:1428] Setting key 2a0c411c9cb6399ab0dd6aef4ac56302
I20260929 14:52:00.888939 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4: Bootstrap starting.
I20260929 14:52:00.891762 22034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:00.892714 22034 log.cc:824] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:00.894809 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4: No bootstrap required, opened a new log
Sep 29 14:52:00 dist-test-slave-5hd1 krb5kdc[21953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:00.899690 22034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } }
I20260929 14:52:00.899950 22034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:00.899996 22034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d5bf8848eb4c038899ef8c763309c4, State: Initialized, Role: FOLLOWER
I20260929 14:52:00.900574 22034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } }
I20260929 14:52:00.900745 22034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:00.900833 22034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:00.900967 22034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:00.902323 22034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } }
I20260929 14:52:00.902730 22034 leader_election.cc:302] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d5bf8848eb4c038899ef8c763309c4; no voters:
I20260929 14:52:00.903122 22034 leader_election.cc:288] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:00.903285 22039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:00.903564 22039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [term 1 LEADER]: Becoming Leader. State: Replica: 81d5bf8848eb4c038899ef8c763309c4, State: Running, Role: LEADER
I20260929 14:52:00.903987 22039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } }
I20260929 14:52:00.904368 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:00.889689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41139 (local address 127.0.98.62:46051)
0929 14:52:00.890202 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:00.890222 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:00.890260 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:52:00.891033 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:00.891050 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:00.891401 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:00.891761 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:00.892615 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:00.893650 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:00.894512 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:00.895033 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:00.895207 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:00.898325 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:00.898331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:00.898343 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:00.898370 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:00.900586 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:00.901210 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:00.901221 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:00.901227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:00.901412 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:00.901786 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:00.901790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:00.901791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:00.902408 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:00.902635 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:00.902941 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 14:52:00.903251 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":166,"threads_started":1}
I20260929 14:52:00.904668 22034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:00.905795 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d5bf8848eb4c038899ef8c763309c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } } }
I20260929 14:52:00.905910 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:00.906333 22044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:00.906281 22041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d5bf8848eb4c038899ef8c763309c4. Latest consensus state: current_term: 1 leader_uuid: "81d5bf8848eb4c038899ef8c763309c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d5bf8848eb4c038899ef8c763309c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46051 } } }
I20260929 14:52:00.906443 22041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:00.908043 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:00.909789 22044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:00.917229 22044 catalog_manager.cc:1347] Generated new cluster ID: 6488f5adfef4456ba5a1eef4d004edd5
I20260929 14:52:00.917302 22044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:00.919332 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:00.910809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33897 (local address 127.0.98.62:46051)
0929 14:52:00.910942 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:00.910947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:00.911286 (+ 339us) server_negotiation.cc:407] Connection header received
0929 14:52:00.911413 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:00.911416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:00.911474 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:00.911571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:00.912384 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:00.912940 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:00.913745 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:00.914013 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:00.914121 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:00.915713 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:00.915717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:00.915719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:00.915751 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:00.917386 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:00.917939 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:00.917943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:00.917945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:00.918028 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:00.918433 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:00.918437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:00.918438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:00.918622 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:00.918733 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:00.919078 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 14:52:00.919183 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:52:00.924057 22044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:00.925034 22044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:00.931978 22044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d5bf8848eb4c038899ef8c763309c4: Generated new TSK 0
I20260929 14:52:00.932677 22044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:00.942170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 21966
W20260929 14:52:00.948774 21951 connection.cc:535] client connection to 127.0.98.62:46051 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46051): Connection reset by peer (error 104)
2026-09-29T14:52:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:01 dist-test-slave-5hd1 krb5kdc[22065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:01 dist-test-slave-5hd1 krb5kdc[22065](info): set up 2 sockets
Sep 29 14:52:01 dist-test-slave-5hd1 krb5kdc[22065](info): commencing operation
krb5kdc: starting...
W20260929 14:52:01.992587 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:02 dist-test-slave-5hd1 krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:02.132256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:02.245886 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:02.246196 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:02.246280 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:02.250383 22078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:02.250481 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:02.250519 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:02.250581 22078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:02.250615 22078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:02.255867 22078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:02.256923 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:02.257912 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:02.266862 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:02.266906 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:02.267117 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:02.267812 22078 server_base.cc:1034] running on GCE node
I20260929 14:52:02.268510 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:02.269603 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:02.270776 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790693522270745 us; error 57 us; skew 500 ppm
Sep 29 14:52:02 dist-test-slave-5hd1 krb5kdc[22065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:02.273756 22078 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:02.274686 22078 webserver.cc:460] Webserver started at http://127.0.98.62:39585/ using document root <none> and password file <none>
I20260929 14:52:02.275238 22078 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:02.275331 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:02.275561 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:02.277587 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "fa8c9d74cefc4de68799be41c0b57a42"
format_stamp: "Formatted at 2026-09-29 14:52:02 on dist-test-slave-5hd1"
server_key: "e236d5dff30cf14d3b11a26a8cea59cb"
server_key_iv: "3569a8df5686758a933d63542648a500"
server_key_version: "encryptionkey@0"
I20260929 14:52:02.278146 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa8c9d74cefc4de68799be41c0b57a42"
format_stamp: "Formatted at 2026-09-29 14:52:02 on dist-test-slave-5hd1"
server_key: "e236d5dff30cf14d3b11a26a8cea59cb"
server_key_iv: "3569a8df5686758a933d63542648a500"
server_key_version: "encryptionkey@0"
I20260929 14:52:02.281963 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 14:52:02.284664 22093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:02.286048 22078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:52:02.286230 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "fa8c9d74cefc4de68799be41c0b57a42"
format_stamp: "Formatted at 2026-09-29 14:52:02 on dist-test-slave-5hd1"
server_key: "e236d5dff30cf14d3b11a26a8cea59cb"
server_key_iv: "3569a8df5686758a933d63542648a500"
server_key_version: "encryptionkey@0"
I20260929 14:52:02.286355 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:02.305783 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:02.308969 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:02.309193 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:02.316740 22078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33677
I20260929 14:52:02.316768 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33677 every 8 connection(s)
I20260929 14:52:02.317898 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:02.319716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22078
I20260929 14:52:02.319892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:02.320288 392 external_mini_cluster.cc:1428] Setting key c81cfff5d926db67113b8840a6c073e1
I20260929 14:52:02.321808 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:02.328744 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: Bootstrap starting.
I20260929 14:52:02.330060 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:02.322074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33759 (local address 127.0.98.62:33677)
0929 14:52:02.322633 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:02.322649 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:02.322691 (+ 42us) server_negotiation.cc:407] Connection header received
0929 14:52:02.323407 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:02.323423 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:02.323798 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:02.324158 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:02.324626 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:02.325641 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:02.326528 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:02.327012 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:02.327187 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:02.328289 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:02.328294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:02.328308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:02.328714 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:02.328768 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:02.328790 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:02.328956 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:02.329154 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 14:52:02.329305 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":192,"threads_started":1}
I20260929 14:52:02.331799 22146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:02.332880 22146 log.cc:824] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:02.333786 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:02.335474 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: No bootstrap required, opened a new log
I20260929 14:52:02.340857 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:02.336089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50321 (local address 127.0.98.62:33677)
0929 14:52:02.336243 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:02.336247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:02.336450 (+ 203us) server_negotiation.cc:407] Connection header received
0929 14:52:02.336605 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:02.336607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:02.336642 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:02.336717 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:02.337071 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:02.337589 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:02.338863 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:02.339120 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:02.339218 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:02.340322 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:02.340325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:02.340327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:02.340338 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:02.340362 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:02.340372 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:02.340454 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:02.340649 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 14:52:02.340706 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:52:02.341007 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260929 14:52:02.341377 22146 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:02.341470 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8c9d74cefc4de68799be41c0b57a42, State: Initialized, Role: FOLLOWER
I20260929 14:52:02.342274 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260929 14:52:02.342451 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:02.342554 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:02.342669 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:02.343878 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260929 14:52:02.344384 22146 leader_election.cc:302] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8c9d74cefc4de68799be41c0b57a42; no voters:
I20260929 14:52:02.344765 22146 leader_election.cc:288] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:02.344918 22156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:02.345247 22156 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [term 1 LEADER]: Becoming Leader. State: Replica: fa8c9d74cefc4de68799be41c0b57a42, State: Running, Role: LEADER
I20260929 14:52:02.345618 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260929 14:52:02.346504 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:02.348352 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa8c9d74cefc4de68799be41c0b57a42. Latest consensus state: current_term: 1 leader_uuid: "fa8c9d74cefc4de68799be41c0b57a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } } }
I20260929 14:52:02.348552 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:02.348883 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa8c9d74cefc4de68799be41c0b57a42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8c9d74cefc4de68799be41c0b57a42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } } }
I20260929 14:52:02.349012 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42 [sys.catalog]: This master's current role is: LEADER
W20260929 14:52:02.352596 22172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:02.352702 22172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:02.352811 22169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:02.353868 22169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:02.361039 22169 catalog_manager.cc:1347] Generated new cluster ID: a17c39d6da3e413c964e349a0f7ccee6
I20260929 14:52:02.361127 22169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:02.381984 22169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:02.383226 22169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:02.389379 22169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa8c9d74cefc4de68799be41c0b57a42: Generated new TSK 0
I20260929 14:52:02.390446 22169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:02.404516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22078
W20260929 14:52:02.410538 22063 connection.cc:535] client connection to 127.0.98.62:33677 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33677): Connection reset by peer (error 104)
2026-09-29T14:52:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:02 dist-test-slave-5hd1 krb5kdc[22177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:02 dist-test-slave-5hd1 krb5kdc[22177](info): set up 2 sockets
Sep 29 14:52:02 dist-test-slave-5hd1 krb5kdc[22177](info): commencing operation
krb5kdc: starting...
W20260929 14:52:03.476840 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:03 dist-test-slave-5hd1 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:03.623652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:03.736899 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:03.737162 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:03.737201 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:03.741222 22190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:03.741287 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:03.741307 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:03.741323 22190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:03.741338 22190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:03.746498 22190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:03.747488 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:03.748477 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:03.757529 22196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:03.757550 22198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:03.757542 22195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:03.757867 22190 server_base.cc:1034] running on GCE node
I20260929 14:52:03.758667 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:03.759756 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:03.760970 22190 hybrid_clock.cc:648] HybridClock initialized: now 1790693523760921 us; error 71 us; skew 500 ppm
Sep 29 14:52:03 dist-test-slave-5hd1 krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:03.769169 22190 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:03.770228 22190 webserver.cc:460] Webserver started at http://127.0.98.62:34997/ using document root <none> and password file <none>
I20260929 14:52:03.770810 22190 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:03.770905 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:03.771152 22190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:03.773355 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "27640ec4a78e48a188a6272dd509f547"
format_stamp: "Formatted at 2026-09-29 14:52:03 on dist-test-slave-5hd1"
server_key: "deb6b3edaf3f3874db4ed4ea282b5b56"
server_key_iv: "df5b0feda14af406ec0b2cd7a7eea4b3"
server_key_version: "encryptionkey@0"
I20260929 14:52:03.773924 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "27640ec4a78e48a188a6272dd509f547"
format_stamp: "Formatted at 2026-09-29 14:52:03 on dist-test-slave-5hd1"
server_key: "deb6b3edaf3f3874db4ed4ea282b5b56"
server_key_iv: "df5b0feda14af406ec0b2cd7a7eea4b3"
server_key_version: "encryptionkey@0"
I20260929 14:52:03.778264 22190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:52:03.781544 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:03.783140 22190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:52:03.783326 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "27640ec4a78e48a188a6272dd509f547"
format_stamp: "Formatted at 2026-09-29 14:52:03 on dist-test-slave-5hd1"
server_key: "deb6b3edaf3f3874db4ed4ea282b5b56"
server_key_iv: "df5b0feda14af406ec0b2cd7a7eea4b3"
server_key_version: "encryptionkey@0"
I20260929 14:52:03.783452 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:03.788293 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:03.791676 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:03.791884 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:03.799629 22190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42743
I20260929 14:52:03.799661 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42743 every 8 connection(s)
I20260929 14:52:03.800684 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:03.804637 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:03.810412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22190
I20260929 14:52:03.810527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:03.810835 392 external_mini_cluster.cc:1428] Setting key f49c99c78515125ef164fec00201717c
I20260929 14:52:03.811579 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: Bootstrap starting.
I20260929 14:52:03.814886 22258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:03.816057 22258 log.cc:824] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:03.816996 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:03.812441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38671 (local address 127.0.98.62:42743)
0929 14:52:03.812911 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:03.812926 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:03.812965 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:52:03.813743 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:03.813762 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:03.814068 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:03.814390 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:03.815551 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:03.815557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:03.815569 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:03.815581 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:03.815655 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:03.815679 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:03.815800 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:03.815966 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:52:03.816172 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":150,"threads_started":1}
I20260929 14:52:03.818620 22258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: No bootstrap required, opened a new log
I20260929 14:52:03.820242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:03.823760 22258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 14:52:03.824110 22258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:03.824193 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:03.822378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51519 (local address 127.0.98.62:42743)
0929 14:52:03.822525 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:03.822529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:03.822738 (+ 209us) server_negotiation.cc:407] Connection header received
0929 14:52:03.822840 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:03.822842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:03.822897 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:03.822997 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:03.823717 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:03.823720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:03.823723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:03.823736 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:03.823768 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:03.823785 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:03.823897 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:03.823978 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 14:52:03.824023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:52:03.824172 22258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27640ec4a78e48a188a6272dd509f547, State: Initialized, Role: FOLLOWER
I20260929 14:52:03.824743 22258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 14:52:03.824923 22258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:03.825016 22258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:03.825161 22258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:03.826337 22258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 14:52:03.826685 22258 leader_election.cc:302] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27640ec4a78e48a188a6272dd509f547; no voters:
I20260929 14:52:03.826982 22258 leader_election.cc:288] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:03.827387 22268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:03.827639 22268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [term 1 LEADER]: Becoming Leader. State: Replica: 27640ec4a78e48a188a6272dd509f547, State: Running, Role: LEADER
I20260929 14:52:03.828025 22268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 14:52:03.828485 22258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:03.829795 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27640ec4a78e48a188a6272dd509f547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260929 14:52:03.829943 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:03.830240 22270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27640ec4a78e48a188a6272dd509f547. Latest consensus state: current_term: 1 leader_uuid: "27640ec4a78e48a188a6272dd509f547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27640ec4a78e48a188a6272dd509f547" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260929 14:52:03.830335 22270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547 [sys.catalog]: This master's current role is: LEADER
W20260929 14:52:03.833932 22283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:03.834033 22283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:03.834128 22277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:03.834957 22277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:03.841485 22277 catalog_manager.cc:1347] Generated new cluster ID: 5b4b1ab0c8ef4035baa77778f99453f4
I20260929 14:52:03.841570 22277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:03.853168 22277 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:03.853605 22277 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:03.870566 22277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27640ec4a78e48a188a6272dd509f547: Generated new TSK 0
I20260929 14:52:03.871464 22277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:03.883011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22190
2026-09-29T14:52:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:03 dist-test-slave-5hd1 krb5kdc[22289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:03 dist-test-slave-5hd1 krb5kdc[22289](info): set up 2 sockets
Sep 29 14:52:03 dist-test-slave-5hd1 krb5kdc[22289](info): commencing operation
krb5kdc: starting...
W20260929 14:52:04.932651 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:05.068529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43731
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:05.198132 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:05.198537 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:05.198629 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:05.203007 22302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:05.203137 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:05.203176 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:05.203219 22302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:05.203253 22302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:05.209592 22302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:05.211098 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:05.212484 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:05.223771 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:05.223757 22307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:05.223910 22302 server_base.cc:1034] running on GCE node
W20260929 14:52:05.223757 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:05.224875 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:05.226189 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:05.227412 22302 hybrid_clock.cc:648] HybridClock initialized: now 1790693525227378 us; error 58 us; skew 500 ppm
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:05.230572 22302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:05.231542 22302 webserver.cc:460] Webserver started at http://127.0.98.62:46571/ using document root <none> and password file <none>
I20260929 14:52:05.232242 22302 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:05.232340 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:05.232583 22302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:05.234690 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "4d264abf35354163aa2ced5767280946"
format_stamp: "Formatted at 2026-09-29 14:52:05 on dist-test-slave-5hd1"
server_key: "add539c4f5e1c67ae831a970ab8a79ce"
server_key_iv: "6f6421754e949edfdef7fa99913c2e99"
server_key_version: "encryptionkey@0"
I20260929 14:52:05.235293 22302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d264abf35354163aa2ced5767280946"
format_stamp: "Formatted at 2026-09-29 14:52:05 on dist-test-slave-5hd1"
server_key: "add539c4f5e1c67ae831a970ab8a79ce"
server_key_iv: "6f6421754e949edfdef7fa99913c2e99"
server_key_version: "encryptionkey@0"
I20260929 14:52:05.239580 22302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 14:52:05.242851 22317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:05.244586 22302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:52:05.244746 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "4d264abf35354163aa2ced5767280946"
format_stamp: "Formatted at 2026-09-29 14:52:05 on dist-test-slave-5hd1"
server_key: "add539c4f5e1c67ae831a970ab8a79ce"
server_key_iv: "6f6421754e949edfdef7fa99913c2e99"
server_key_version: "encryptionkey@0"
I20260929 14:52:05.244875 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:05.272298 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:05.275839 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:05.276091 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:05.284451 22302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43731
I20260929 14:52:05.284541 22369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43731 every 8 connection(s)
I20260929 14:52:05.285481 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:05.286588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22302
I20260929 14:52:05.286751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:05.287110 392 external_mini_cluster.cc:1428] Setting key 87ff13eedfcbec50c21b835a81a053e4
I20260929 14:52:05.289614 22370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:05.296367 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946: Bootstrap starting.
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:05.299223 22370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:05.300235 22370 log.cc:824] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:05.302935 22370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946: No bootstrap required, opened a new log
I20260929 14:52:05.303005 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:05.288926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56931 (local address 127.0.98.62:43731)
0929 14:52:05.289488 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:05.289505 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:05.289545 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:52:05.290306 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:05.290324 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:05.290706 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:05.291067 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:05.291871 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:05.292960 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:05.293803 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:05.294278 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:05.294441 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:05.297553 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:05.297559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:05.297572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:05.297609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:05.299606 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:05.300299 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:05.300304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:05.300309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:05.300477 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:05.300827 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:05.300830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:05.300832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:05.301371 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:05.301526 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:05.301885 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 14:52:05.302166 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":204,"threads_started":1}
I20260929 14:52:05.306432 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:05.307686 22370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } }
I20260929 14:52:05.308060 22370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:05.308246 22370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d264abf35354163aa2ced5767280946, State: Initialized, Role: FOLLOWER
I20260929 14:52:05.308897 22370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } }
I20260929 14:52:05.309165 22370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:05.309279 22370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:05.309429 22370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:05.310655 22370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } }
I20260929 14:52:05.311146 22370 leader_election.cc:302] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d264abf35354163aa2ced5767280946; no voters:
I20260929 14:52:05.311587 22370 leader_election.cc:288] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:05.311688 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:05.311916 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [term 1 LEADER]: Becoming Leader. State: Replica: 4d264abf35354163aa2ced5767280946, State: Running, Role: LEADER
I20260929 14:52:05.312413 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } }
I20260929 14:52:05.313329 22370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:05.313983 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d264abf35354163aa2ced5767280946. Latest consensus state: current_term: 1 leader_uuid: "4d264abf35354163aa2ced5767280946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } } }
I20260929 14:52:05.314198 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:05.314769 22387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:05.314903 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d264abf35354163aa2ced5767280946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d264abf35354163aa2ced5767280946" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43731 } } }
I20260929 14:52:05.315114 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:05.318365 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:05.308779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47731 (local address 127.0.98.62:43731)
0929 14:52:05.308956 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:05.308961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:05.309057 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:52:05.309228 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:05.309232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:05.309285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:05.309375 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:05.310029 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:05.310557 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:05.311499 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:05.311881 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:05.312002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:05.313583 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:05.313587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:05.313589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:05.313609 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:05.315557 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:05.316391 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:05.316397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:05.316401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:05.316513 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:05.316868 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:05.316870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:05.316873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:05.317122 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:05.317951 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:05.318030 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 14:52:05.318167 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:52:05.318632 22387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:05.325033 22387 catalog_manager.cc:1347] Generated new cluster ID: 9cd5144be0174ff196dad95c8ed63df9
I20260929 14:52:05.325099 22387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:05.356017 22387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:05.357059 22387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:05.365350 22387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d264abf35354163aa2ced5767280946: Generated new TSK 0
I20260929 14:52:05.366022 22387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:05.375064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22302
2026-09-29T14:52:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22401](info): set up 2 sockets
Sep 29 14:52:05 dist-test-slave-5hd1 krb5kdc[22401](info): commencing operation
krb5kdc: starting...
W20260929 14:52:06.416441 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:06 dist-test-slave-5hd1 krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:06.552742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:06.672140 22414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:06.672552 22414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:06.672642 22414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:06.676894 22414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:06.676985 22414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:06.677022 22414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:06.677081 22414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:06.677112 22414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:06.682538 22414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:06.683670 22414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:06.684770 22414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:06.693985 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:06.694075 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:06.693986 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:06.694512 22414 server_base.cc:1034] running on GCE node
I20260929 14:52:06.695087 22414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:06.696298 22414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:06.697499 22414 hybrid_clock.cc:648] HybridClock initialized: now 1790693526697481 us; error 38 us; skew 500 ppm
Sep 29 14:52:06 dist-test-slave-5hd1 krb5kdc[22401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:06.700444 22414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:06.701416 22414 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260929 14:52:06.701969 22414 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:06.702034 22414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:06.702203 22414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:06.704348 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "fb1724092a8246018fedc152a38ee859"
format_stamp: "Formatted at 2026-09-29 14:52:06 on dist-test-slave-5hd1"
server_key: "b6d9b7932188b7592b2dde68e8a0754e"
server_key_iv: "4aff89dd4e6741f35cf0f5ed2bc5132f"
server_key_version: "encryptionkey@0"
I20260929 14:52:06.704888 22414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb1724092a8246018fedc152a38ee859"
format_stamp: "Formatted at 2026-09-29 14:52:06 on dist-test-slave-5hd1"
server_key: "b6d9b7932188b7592b2dde68e8a0754e"
server_key_iv: "4aff89dd4e6741f35cf0f5ed2bc5132f"
server_key_version: "encryptionkey@0"
I20260929 14:52:06.709324 22414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:52:06.712374 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:06.713893 22414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:52:06.714049 22414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "fb1724092a8246018fedc152a38ee859"
format_stamp: "Formatted at 2026-09-29 14:52:06 on dist-test-slave-5hd1"
server_key: "b6d9b7932188b7592b2dde68e8a0754e"
server_key_iv: "4aff89dd4e6741f35cf0f5ed2bc5132f"
server_key_version: "encryptionkey@0"
I20260929 14:52:06.714176 22414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:06.732563 22414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:06.735663 22414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:06.735894 22414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:06.743697 22414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41673
I20260929 14:52:06.743870 22481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41673 every 8 connection(s)
I20260929 14:52:06.744786 22414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:06.748327 22482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:06.754163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22414
I20260929 14:52:06.754325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:06.754634 392 external_mini_cluster.cc:1428] Setting key 9cf39db90ba29d730107f442c28a5f64
I20260929 14:52:06.755319 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859: Bootstrap starting.
I20260929 14:52:06.758244 22482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:06.759099 22482 log.cc:824] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:06.761408 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859: No bootstrap required, opened a new log
I20260929 14:52:06.765535 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:06.756507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38991 (local address 127.0.98.62:41673)
0929 14:52:06.757146 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:06.757168 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:06.757215 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:52:06.758124 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:06.758144 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:06.758615 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:06.759048 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:06.759541 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:06.760860 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:06.761909 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:06.762440 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:06.762631 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:06.763712 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:06.763717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:06.763730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:06.764130 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:06.764236 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:06.764274 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:06.764382 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:06.764532 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 14:52:06.764755 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":207,"threads_started":1}
I20260929 14:52:06.765889 22482 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } }
I20260929 14:52:06.766191 22482 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:06.766240 22482 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1724092a8246018fedc152a38ee859, State: Initialized, Role: FOLLOWER
I20260929 14:52:06.766760 22482 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } }
I20260929 14:52:06.766911 22482 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:06.767005 22482 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:06.767133 22482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:06.768467 22482 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } }
I20260929 14:52:06.768906 22482 leader_election.cc:302] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb1724092a8246018fedc152a38ee859; no voters:
I20260929 14:52:06.769090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:06.769348 22482 leader_election.cc:288] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:06.769423 22487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:06.769634 22487 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [term 1 LEADER]: Becoming Leader. State: Replica: fb1724092a8246018fedc152a38ee859, State: Running, Role: LEADER
I20260929 14:52:06.769992 22487 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } }
I20260929 14:52:06.770808 22482 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:06.772038 22489 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb1724092a8246018fedc152a38ee859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } } }
I20260929 14:52:06.772313 22489 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:06.772033 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb1724092a8246018fedc152a38ee859. Latest consensus state: current_term: 1 leader_uuid: "fb1724092a8246018fedc152a38ee859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1724092a8246018fedc152a38ee859" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41673 } } }
I20260929 14:52:06.772704 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:06.773262 22501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:06.776865 22501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:06.777391 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:06.772970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47929 (local address 127.0.98.62:41673)
0929 14:52:06.773075 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:06.773081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:06.773098 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:52:06.773140 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:06.773143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:06.773188 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:06.773293 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:06.773654 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:06.774390 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:06.775331 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:06.775645 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:06.775749 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:06.776823 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:06.776827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:06.776830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:06.776844 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:06.776877 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:06.776892 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:06.776984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:06.777171 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:52:06.777246 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 14:52:06.782860 22501 catalog_manager.cc:1347] Generated new cluster ID: 1b018785afe14913b90c2920ae60c32e
I20260929 14:52:06.782928 22501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:06.795130 22501 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:06.796293 22501 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:06.802124 22501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb1724092a8246018fedc152a38ee859: Generated new TSK 0
I20260929 14:52:06.802860 22501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:06.811947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22414
2026-09-29T14:52:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:06 dist-test-slave-5hd1 krb5kdc[22513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:06 dist-test-slave-5hd1 krb5kdc[22513](info): set up 2 sockets
Sep 29 14:52:06 dist-test-slave-5hd1 krb5kdc[22513](info): commencing operation
krb5kdc: starting...
W20260929 14:52:07.856667 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:07 dist-test-slave-5hd1 krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:07.991418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:08.122207 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:08.122561 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:08.122610 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:08.127032 22526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:08.127126 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:08.127146 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:08.127166 22526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:08.127182 22526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:08.133464 22526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:08.134929 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:08.136312 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:08.146298 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:08.146298 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:08.146461 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:08.146770 22526 server_base.cc:1034] running on GCE node
I20260929 14:52:08.147586 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:08.149057 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:08.150255 22526 hybrid_clock.cc:648] HybridClock initialized: now 1790693528150240 us; error 37 us; skew 500 ppm
Sep 29 14:52:08 dist-test-slave-5hd1 krb5kdc[22513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:08.153702 22526 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:08.154958 22526 webserver.cc:460] Webserver started at http://127.0.98.62:46527/ using document root <none> and password file <none>
I20260929 14:52:08.155663 22526 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:08.155764 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:08.156068 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:08.158630 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "787893324c2248d7b728f41b0b7c237d"
format_stamp: "Formatted at 2026-09-29 14:52:08 on dist-test-slave-5hd1"
server_key: "611e4ed8343d974a296161b94a311fd3"
server_key_iv: "a3bf5e06c36eb5e3c2b5445079fe7e1c"
server_key_version: "encryptionkey@0"
I20260929 14:52:08.159286 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "787893324c2248d7b728f41b0b7c237d"
format_stamp: "Formatted at 2026-09-29 14:52:08 on dist-test-slave-5hd1"
server_key: "611e4ed8343d974a296161b94a311fd3"
server_key_iv: "a3bf5e06c36eb5e3c2b5445079fe7e1c"
server_key_version: "encryptionkey@0"
I20260929 14:52:08.164052 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 14:52:08.167081 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:08.168743 22526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:52:08.168919 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "787893324c2248d7b728f41b0b7c237d"
format_stamp: "Formatted at 2026-09-29 14:52:08 on dist-test-slave-5hd1"
server_key: "611e4ed8343d974a296161b94a311fd3"
server_key_iv: "a3bf5e06c36eb5e3c2b5445079fe7e1c"
server_key_version: "encryptionkey@0"
I20260929 14:52:08.169073 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:08.174566 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:08.178383 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:08.178629 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:08.187603 22526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35663
I20260929 14:52:08.187623 22593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35663 every 8 connection(s)
I20260929 14:52:08.188839 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:08.189388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22526
I20260929 14:52:08.189538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:08.189857 392 external_mini_cluster.cc:1428] Setting key 4b3464f21e17bd60034b4b93601b35f9
I20260929 14:52:08.192768 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:08.195796 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:08.191396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59143 (local address 127.0.98.62:35663)
0929 14:52:08.191940 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:08.191959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:08.192006 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:52:08.192822 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:08.192845 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:08.193259 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:08.193638 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:08.194507 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:08.194517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:08.194529 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:08.194540 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:08.194601 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:08.194622 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:08.194791 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:08.194865 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 14:52:08.195026 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":165,"threads_started":1}
I20260929 14:52:08.199559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:08.201459 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d: Bootstrap starting.
I20260929 14:52:08.204113 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:08.202065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33987 (local address 127.0.98.62:35663)
0929 14:52:08.202221 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:08.202226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:08.202494 (+ 268us) server_negotiation.cc:407] Connection header received
0929 14:52:08.202595 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:08.202599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:08.202634 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:08.202721 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:08.203626 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:08.203629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:08.203631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:52:08.203641 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:08.203669 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:52:08.203679 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:08.203768 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:08.203898 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 14:52:08.203932 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 14:52:08.205176 22594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:08.206115 22594 log.cc:824] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:08.208812 22594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d: No bootstrap required, opened a new log
I20260929 14:52:08.213531 22594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } }
I20260929 14:52:08.214087 22594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:52:08.214207 22594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787893324c2248d7b728f41b0b7c237d, State: Initialized, Role: FOLLOWER
I20260929 14:52:08.214941 22594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } }
I20260929 14:52:08.215179 22594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:52:08.215276 22594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:52:08.215422 22594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:08.217000 22594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } }
I20260929 14:52:08.217475 22594 leader_election.cc:302] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787893324c2248d7b728f41b0b7c237d; no voters:
I20260929 14:52:08.217895 22594 leader_election.cc:288] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:52:08.218104 22604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:08.218530 22604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [term 1 LEADER]: Becoming Leader. State: Replica: 787893324c2248d7b728f41b0b7c237d, State: Running, Role: LEADER
I20260929 14:52:08.218956 22604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } }
I20260929 14:52:08.219584 22594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:08.221006 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787893324c2248d7b728f41b0b7c237d. Latest consensus state: current_term: 1 leader_uuid: "787893324c2248d7b728f41b0b7c237d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } }
I20260929 14:52:08.221177 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:08.221622 22613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:08.221516 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787893324c2248d7b728f41b0b7c237d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787893324c2248d7b728f41b0b7c237d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35663 } } }
I20260929 14:52:08.222504 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:08.224979 22613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:08.230942 22613 catalog_manager.cc:1347] Generated new cluster ID: 56d758d2d1314edea824dd7d1c18766e
I20260929 14:52:08.231024 22613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:08.245782 22613 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:08.246292 22613 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:08.253533 22613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787893324c2248d7b728f41b0b7c237d: Generated new TSK 0
I20260929 14:52:08.254575 22613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:52:08.262753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22526
2026-09-29T14:52:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1451 ms)
[----------] 18 tests from AuthTokenIssuingTest (26429 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:08 dist-test-slave-5hd1 krb5kdc[22625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:08 dist-test-slave-5hd1 krb5kdc[22625](info): set up 2 sockets
Sep 29 14:52:08 dist-test-slave-5hd1 krb5kdc[22625](info): commencing operation
krb5kdc: starting...
W20260929 14:52:09.308800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:09.444087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:09.561061 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:09.561424 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:09.561476 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:09.561554 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:09.565954 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:09.566073 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:09.566114 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:09.566154 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:09.566188 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:09.571466 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:09.572629 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:09.573946 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:52:09.583088 22638 server_base.cc:1034] running on GCE node
W20260929 14:52:09.583020 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:09.583041 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:09.583020 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:09.583922 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:09.585129 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:09.586347 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790693529586328 us; error 45 us; skew 500 ppm
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:09.596884 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:09.597908 22638 webserver.cc:460] Webserver started at http://127.0.98.62:44495/ using document root <none> and password file <none>
I20260929 14:52:09.598495 22638 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:09.598560 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:09.598735 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:09.600901 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "b25915fef0c44fb2bb590cb1c2a15520"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "f7b13f20b96f058b9c1f65248edb2999"
server_key_iv: "1b6afbbf312d94d670c0bdb14960deff"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.601441 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "b25915fef0c44fb2bb590cb1c2a15520"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "f7b13f20b96f058b9c1f65248edb2999"
server_key_iv: "1b6afbbf312d94d670c0bdb14960deff"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.605820 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:52:09.608610 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:09.610050 22638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:52:09.610176 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "b25915fef0c44fb2bb590cb1c2a15520"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "f7b13f20b96f058b9c1f65248edb2999"
server_key_iv: "1b6afbbf312d94d670c0bdb14960deff"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.610333 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:09.622753 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:09.625990 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:09.626219 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:09.633836 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33811
I20260929 14:52:09.634215 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33811 every 8 connection(s)
I20260929 14:52:09.634850 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:09.638474 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:09.641944 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22638
I20260929 14:52:09.642087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:09.642374 392 external_mini_cluster.cc:1428] Setting key dd9b150a93452fa1b6354f0ea4f103b3
I20260929 14:52:09.643235 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:09.660696 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.645955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.0.98.62:33811)
0929 14:52:09.646621 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:09.646643 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:09.646790 (+ 147us) server_negotiation.cc:407] Connection header received
0929 14:52:09.648001 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:09.648006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:09.648264 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:09.648475 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:09.649402 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.650556 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.651641 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.652144 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.652347 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.655216 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:09.655223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:09.655245 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:09.655267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:09.657070 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.657623 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.657631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.657633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.657749 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.658086 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.658089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.658091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.658556 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:09.658704 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:09.659105 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 14:52:09.659419 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":565,"spinlock_wait_cycles":97664,"thread_start_us":248,"threads_started":1}
I20260929 14:52:09.660738 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.645954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33811 (local address 127.0.0.1:46502)
0929 14:52:09.646578 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.646636 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:09.647642 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:09.648475 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:09.648480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:09.648985 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:09.649250 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.649264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.650691 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.650694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.651515 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.651525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.652347 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.652351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.652506 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.653253 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:09.653272 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:09.655046 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:09.657206 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.657217 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.657228 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.657476 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.657854 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.657857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.657859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.657979 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.658681 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:09.658688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:09.658843 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 14:52:09.659046 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 14:52:09.659469 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":111,"thread_start_us":158,"threads_started":1}
I20260929 14:52:09.663827 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:09.665011 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.664659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50564)
0929 14:52:09.664824 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.664912 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 14:52:09.665277 22656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
I20260929 14:52:09.666333 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.666117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50580)
0929 14:52:09.666226 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.666275 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 14:52:09.666505 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37273: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:52:09.668526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37273
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:52:09.715847 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } attempt: 1
I20260929 14:52:09.717072 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.716669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50582)
0929 14:52:09.716880 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.716970 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 14:52:09.717923 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.717696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50586)
0929 14:52:09.717806 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.717864 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:52:09.718101 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37273: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
W20260929 14:52:09.778117 22713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:09.778475 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:09.778527 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:09.778601 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:09.782698 22713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:09.782785 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:09.782823 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:09.782887 22713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:09.782922 22713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:09.786262 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } attempt: 2
I20260929 14:52:09.787418 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.786977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50598)
0929 14:52:09.787210 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.787300 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 14:52:09.788316 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.788051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50614)
0929 14:52:09.788164 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.788253 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 14:52:09.788467 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37273: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37273: connect: Connection refused (error 111)
I20260929 14:52:09.788719 22713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:09.789813 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:09.790804 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:09.799722 22718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:09.799739 22721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:09.799961 22713 server_base.cc:1034] running on GCE node
W20260929 14:52:09.799793 22719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:09.800925 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:09.802383 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:09.803617 22713 hybrid_clock.cc:648] HybridClock initialized: now 1790693529803546 us; error 88 us; skew 500 ppm
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:09.807204 22713 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:52:09.808514 22713 webserver.cc:460] Webserver started at http://127.0.98.61:41801/ using document root <none> and password file <none>
I20260929 14:52:09.809250 22713 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:09.809362 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:09.809648 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:09.812263 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data/instance:
uuid: "4874aa4e2913481eb32959a53bf7f332"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "96293473196e14ad9d6feae42ba62c7e"
server_key_iv: "8404b052aa0205a4d1743cec84e71a0c"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.812943 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal/instance:
uuid: "4874aa4e2913481eb32959a53bf7f332"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "96293473196e14ad9d6feae42ba62c7e"
server_key_iv: "8404b052aa0205a4d1743cec84e71a0c"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.817731 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 14:52:09.821583 22728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:09.823350 22713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:52:09.823554 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "4874aa4e2913481eb32959a53bf7f332"
format_stamp: "Formatted at 2026-09-29 14:52:09 on dist-test-slave-5hd1"
server_key: "96293473196e14ad9d6feae42ba62c7e"
server_key_iv: "8404b052aa0205a4d1743cec84e71a0c"
server_key_version: "encryptionkey@0"
I20260929 14:52:09.823711 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:09.844389 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:09.848405 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:09.848652 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:09.857406 22713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37273
I20260929 14:52:09.857750 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37273 every 8 connection(s)
I20260929 14:52:09.858626 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:52:09.861877 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:09.866150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22713
I20260929 14:52:09.865785 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:09.866285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-1/wal/instance
I20260929 14:52:09.866600 392 external_mini_cluster.cc:1428] Setting key bc031e5933443e87b745c0ce018c0654
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:09.879649 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } attempt: 3
I20260929 14:52:09.883421 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.868434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.98.62:33811)
0929 14:52:09.868679 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:09.868686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:09.869802 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 14:52:09.870959 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:09.870962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:09.871042 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:09.871194 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:09.872665 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.873560 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.874880 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.875151 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.875244 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.878864 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:09.878871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:09.878877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:09.878918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:09.880826 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.881593 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.881597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.881600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.881693 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.882114 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.882117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.882119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.882326 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:09.882441 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:09.883082 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 14:52:09.883216 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 14:52:09.884618 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.868944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33811 (local address 127.0.0.1:46516)
0929 14:52:09.869603 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.869664 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:09.870723 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:09.871405 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:09.871412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:09.871830 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:09.872497 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.872512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.873719 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.873721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.874743 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.874760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.875307 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.875310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.875483 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.876231 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:09.876261 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:09.878659 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:09.880972 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.880982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.881006 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.881440 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.881827 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.881830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.881831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.881988 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.882439 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:09.882446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:09.882809 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 14:52:09.883004 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 14:52:09.883530 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":103,"thread_start_us":205,"threads_started":1}
I20260929 14:52:09.886268 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:52:09 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:52:09.894025 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.880535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50626)
0929 14:52:09.880677 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.880695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:09.880787 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:09.881612 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:09.881617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:09.881843 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:09.882021 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.882031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.883723 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.883731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.884833 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.884850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.885826 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.885831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.885977 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.886527 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:09.886542 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:09.888293 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:09.890968 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.890975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.890980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.891245 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.891648 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.891651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.891653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.891754 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.893323 (+ 1569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:09.893327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:09.893469 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 14:52:09.893625 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 14:52:09.893816 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 14:52:09.894188 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.880739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.0.98.61:37273)
0929 14:52:09.881149 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:09.881161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:09.881195 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:52:09.881246 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:09.881251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:09.881435 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:09.881611 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:09.882171 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.883556 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.885012 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.885686 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.885824 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.888448 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:09.888465 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:09.888480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:09.888503 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:09.890849 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.891424 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.891435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.891440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.891548 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.892765 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.892768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.892770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.893181 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:09.893358 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:09.893725 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 14:52:09.893902 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":1280,"thread_start_us":164,"threads_started":1}
I20260929 14:52:09.896610 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:09.897328 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.886949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50640)
0929 14:52:09.887385 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.887403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:09.887479 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:09.887921 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:09.887927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:09.888121 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:09.888304 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.888310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.889265 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.889270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.890151 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:09.890165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.890654 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.890658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:09.890789 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.891217 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:09.891233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:09.892958 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:09.895581 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.895584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.895586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.895775 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.896043 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:09.896046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:09.896047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:09.896145 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:09.896705 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:09.896707 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:09.896830 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 14:52:09.896947 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 14:52:09.897110 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":23,"spinlock_wait_cycles":5376}
I20260929 14:52:09.897563 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.897229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36888)
0929 14:52:09.897390 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.897465 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:52:09.897758 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.887287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.98.61:37273)
0929 14:52:09.887624 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:09.887632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:09.887649 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:52:09.887699 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:09.887704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:09.887750 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:09.887859 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:09.888458 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.889138 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.890283 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:09.890521 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:09.890656 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:09.893109 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:09.893112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:09.893114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:09.893149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:09.895476 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.895872 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.895878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.895880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.895952 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:09.896310 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:09.896315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:09.896318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:09.896579 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:09.896698 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:09.897069 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 14:52:09.897195 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":246,"spinlock_wait_cycles":5632,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:52:09.898738 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:09.898672 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.898330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36892)
0929 14:52:09.898547 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.898604 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260929 14:52:09.899210 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:09.899374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45381
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:52:09.899570 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.899212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36906)
0929 14:52:09.899381 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.899474 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:52:09.899776 22731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:09.900830 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.900549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36918)
0929 14:52:09.900687 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.900751 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 14:52:09.900978 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:09.930411 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 1
I20260929 14:52:09.931600 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.931156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36926)
0929 14:52:09.931380 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.931486 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 14:52:09.932438 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.932156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36936)
0929 14:52:09.932318 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.932374 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 14:52:09.932590 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:09.950323 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 1
I20260929 14:52:09.951494 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.951071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36940)
0929 14:52:09.951283 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.951381 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 14:52:09.952387 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:09.952144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36950)
0929 14:52:09.952279 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:09.952333 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 14:52:09.952574 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:10.007774 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 2
I20260929 14:52:10.008899 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.008411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36956)
0929 14:52:10.008661 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.008782 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 14:52:10.009984 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.009733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36964)
0929 14:52:10.009858 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.009921 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 14:52:10.010140 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
I20260929 14:52:10.020816 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 2
I20260929 14:52:10.021951 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.021497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36978)
0929 14:52:10.021718 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.021820 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 14:52:10.022840 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.022610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:36994)
0929 14:52:10.022724 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.022781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 14:52:10.023037 22781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45381: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45381: connect: Connection refused (error 111)
W20260929 14:52:10.024336 22789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:10.024614 22789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:10.024686 22789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:10.024792 22789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:10.029139 22789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:10.029194 22789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:10.029213 22789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:10.029229 22789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:10.029245 22789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:10.034456 22789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33811,127.0.98.61:37273,127.0.98.60:45381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:10.035538 22789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:10.036644 22789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:10.045712 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:10.045727 22794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:10.045764 22789 server_base.cc:1034] running on GCE node
W20260929 14:52:10.045991 22795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:10.046583 22789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:10.047713 22789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:10.048906 22789 hybrid_clock.cc:648] HybridClock initialized: now 1790693530048865 us; error 67 us; skew 500 ppm
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:10.051811 22789 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:52:10.052793 22789 webserver.cc:460] Webserver started at http://127.0.98.60:38215/ using document root <none> and password file <none>
I20260929 14:52:10.053320 22789 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:10.053388 22789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:10.053555 22789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:10.055514 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data/instance:
uuid: "b5837ee2faef4cd49b63a11715bebdd7"
format_stamp: "Formatted at 2026-09-29 14:52:10 on dist-test-slave-5hd1"
server_key: "2274a04866d5ef72f9976f915f42deec"
server_key_iv: "072b9ce995ec5b18f5d779cb25ae0545"
server_key_version: "encryptionkey@0"
I20260929 14:52:10.056039 22789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal/instance:
uuid: "b5837ee2faef4cd49b63a11715bebdd7"
format_stamp: "Formatted at 2026-09-29 14:52:10 on dist-test-slave-5hd1"
server_key: "2274a04866d5ef72f9976f915f42deec"
server_key_iv: "072b9ce995ec5b18f5d779cb25ae0545"
server_key_version: "encryptionkey@0"
I20260929 14:52:10.059988 22789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 14:52:10.062706 22804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:10.064298 22789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:52:10.064412 22789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "b5837ee2faef4cd49b63a11715bebdd7"
format_stamp: "Formatted at 2026-09-29 14:52:10 on dist-test-slave-5hd1"
server_key: "2274a04866d5ef72f9976f915f42deec"
server_key_iv: "072b9ce995ec5b18f5d779cb25ae0545"
server_key_version: "encryptionkey@0"
I20260929 14:52:10.064502 22789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:10.088110 22789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:10.091370 22789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:10.091537 22789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:10.099318 22789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45381
I20260929 14:52:10.099694 22856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45381 every 8 connection(s)
I20260929 14:52:10.100389 22789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:52:10.103863 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:10.108280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 22789
I20260929 14:52:10.108438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790693238534145-392-0/minicluster-data/master-2/wal/instance
I20260929 14:52:10.108752 392 external_mini_cluster.cc:1428] Setting key 085e8a624cffc558d3bd45bb7568f4c6
I20260929 14:52:10.108842 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:10.109316 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 3
I20260929 14:52:10.114253 22781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } attempt: 3
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:52:10.124890 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.111528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.0.98.62:33811)
0929 14:52:10.111650 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.111654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.112056 (+ 402us) server_negotiation.cc:407] Connection header received
0929 14:52:10.112623 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.112627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.112685 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.112774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.114043 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.114841 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.116759 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117063 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117130 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.120675 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.120679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.120682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.120723 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.122771 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123349 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.123354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.123359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.123429 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123801 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.123804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.123806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.124066 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.124148 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.124647 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.124750 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:52:10.124890 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.110059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:37002)
0929 14:52:10.110306 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.110324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:10.110436 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:10.112747 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:10.112753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:10.113030 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:10.113142 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.113151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.115127 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.115133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.115963 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.115973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117350 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.117435 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.118066 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:10.118105 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:10.119640 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:10.122891 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.122893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.122895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.123150 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.123534 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.123539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.123634 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.124364 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:10.124367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:10.124449 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 14:52:10.124539 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 14:52:10.124691 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 14:52:10.126499 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.111447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33811 (local address 127.0.0.1:46532)
0929 14:52:10.111890 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.111946 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:10.112399 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:10.112983 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:10.112990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:10.113545 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:10.113796 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.113813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.115365 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.115369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.116630 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.116638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117353 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.117457 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.118246 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:10.118269 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:10.120501 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:10.122889 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.122895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.122905 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.123242 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.123535 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.123540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.123688 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.124248 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:10.124254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:10.124400 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 14:52:10.125100 (+ 700us) client_negotiation.cc:240] Negotiation successful
0929 14:52:10.125460 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":76,"spinlock_wait_cycles":25984,"thread_start_us":99,"threads_started":1}
I20260929 14:52:10.126513 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.111116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.0.98.60:45381)
0929 14:52:10.111553 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.111566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.111585 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:52:10.112023 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.112051 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.112470 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.112719 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.113307 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.114961 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.116098 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.116995 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117156 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.119812 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.119819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.119833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.119860 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.122771 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123295 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.123303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.123305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.123432 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.123745 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.123748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.123751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.124251 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.125261 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.125362 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.125515 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":128896,"thread_start_us":175,"threads_started":1}
I20260929 14:52:10.127394 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.114821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:37012)
0929 14:52:10.115098 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.115117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:10.115197 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:10.115536 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:10.115540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:10.115760 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:10.115902 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.115910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117263 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.118697 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.118708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.119483 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.119488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.119604 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.120138 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:10.120157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:10.122715 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:10.125423 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.125427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.125429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.125729 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.126108 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.126114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.126116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.126212 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.126757 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:10.126762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:10.126882 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 14:52:10.126971 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:52:10.127133 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 14:52:10.128295 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.114892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.0.98.60:45381)
0929 14:52:10.115251 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.115262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.115277 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:52:10.115312 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.115315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.115370 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.116029 (+ 659us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.116200 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117110 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.118891 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.119309 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.119884 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.122900 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.122907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.122914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.122946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.125281 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.125866 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.125870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.125872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.125983 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.126340 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.126355 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.126356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.126609 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.127759 (+ 1150us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.127842 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.127990 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260929 14:52:10.128594 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:10.129393 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.110519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.0.98.60:45381)
0929 14:52:10.110948 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.110969 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.111009 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:52:10.112023 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.112050 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.112470 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.113340 (+ 870us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.113684 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.115286 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.116771 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.117157 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.117750 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.123011 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.123015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.123017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.123037 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.126220 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.127360 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.127363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.127365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.127685 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.128053 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.128056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.128058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.128592 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.128925 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.129056 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.129186 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":495,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":15360,"thread_start_us":147,"threads_started":1}
I20260929 14:52:10.131342 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: Bootstrap starting.
I20260929 14:52:10.134107 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:10.135042 22706 log.cc:824] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:10.136081 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: Bootstrap starting.
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:52:10.140749 22781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:10.142045 22781 log.cc:824] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:10.142527 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.130698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.0.98.62:33811)
0929 14:52:10.130809 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.130813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.130829 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:52:10.130985 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.130988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.131030 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.131124 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.131815 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.132543 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.133737 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.134087 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.134214 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.137749 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.137753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.137756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.137788 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.140016 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.140692 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.140696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.140698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.140794 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.141253 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.141257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.141259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.141552 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.141760 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.142182 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.142342 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:52:10.144976 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: No bootstrap required, opened a new log
I20260929 14:52:10.145188 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.132901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.98.61:37273)
0929 14:52:10.133021 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.133025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.133872 (+ 847us) server_negotiation.cc:407] Connection header received
0929 14:52:10.134072 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.134077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.134138 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.134195 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.134924 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.135710 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.137257 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.137666 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.137766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.140523 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.140526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.140529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.140565 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.142599 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.143625 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.143629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.143634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.143729 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.144074 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.144078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.144079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.144424 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.144504 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.144849 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.144994 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:52:10.145206 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.132666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37273 (local address 127.0.0.1:50644)
0929 14:52:10.133794 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.133811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:10.133951 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:10.134455 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:10.134458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:10.134651 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:10.134776 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.134783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.135856 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.135861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.137082 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.137095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.137950 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.137954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.138075 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.138610 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:10.138626 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:10.140343 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:10.143205 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.143208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.143210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.143500 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.143848 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.143851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.143853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.143940 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.144583 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:10.144586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:10.144683 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 14:52:10.144796 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 14:52:10.144959 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049}
I20260929 14:52:10.146425 22857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:10.147589 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: No bootstrap required, opened a new log
I20260929 14:52:10.149027 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.149312 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25915fef0c44fb2bb590cb1c2a15520, State: Initialized, Role: FOLLOWER
I20260929 14:52:10.149646 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.151083 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:10.152197 22781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.152456 22781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4874aa4e2913481eb32959a53bf7f332, State: Initialized, Role: FOLLOWER
I20260929 14:52:10.152833 22781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:52:10.154804 22781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 14:52:10 dist-test-slave-5hd1 krb5kdc[22625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:52:10.157564 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.146995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45381 (local address 127.0.0.1:37020)
0929 14:52:10.147188 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:10.147203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:10.147277 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:10.148648 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:10.148651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:10.148822 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:10.148907 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.148913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.149952 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.149955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.150736 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:10.150746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.151480 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.151483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:10.151562 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.152105 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:10.152119 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:10.153904 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:10.155964 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.155967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.155971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.156225 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.156519 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:10.156521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:10.156523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:10.156600 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:10.157043 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:10.157045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:10.157121 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 14:52:10.157213 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:52:10.157365 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 14:52:10.158311 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.147067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.0.98.60:45381)
0929 14:52:10.148349 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.148354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.148369 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:52:10.148413 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.148416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.148461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.148547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.149021 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.149819 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.150861 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.151250 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.151333 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.154049 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.154053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.154056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.154078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.155841 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.156331 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.156337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.156340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.156431 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.156684 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.156686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.156688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.156918 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.157010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.157946 (+ 936us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.158089 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201}
W20260929 14:52:10.162432 22877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 14:52:10.162643 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: Bootstrap starting.
I20260929 14:52:10.164654 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.164846 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.166249 22857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:10.167289 22857 log.cc:824] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:10.160115 22638 master_runner.cc:186] Error getting master registration for 127.0.98.61:37273: OK, instance_id { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" instance_seqno: 1790693529854984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:52:10.168358 22638 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 14:52:10.168738 22877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:10.154945 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.169147 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.170292 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.144651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.98.61:37273)
0929 14:52:10.144791 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.144796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.148422 (+ 3626us) server_negotiation.cc:407] Connection header received
0929 14:52:10.148605 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.148608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.148658 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.148706 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.149773 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.150530 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.151828 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.152261 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.152348 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.156854 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.156858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.156861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.156895 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.167145 (+ 10250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.168134 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.168139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.168141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.168313 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.168673 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.168677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.168678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.168942 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.169144 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.169975 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.170125 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 14:52:10.171979 22890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:10.172098 22890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:10.172433 22857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: No bootstrap required, opened a new log
I20260929 14:52:10.173789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:10.171073 22713 master_runner.cc:186] Error getting master registration for 127.0.98.60:45381: OK, instance_id { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" instance_seqno: 1790693530097017 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:52:10.174710 22713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:52:10.175231 22680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b25915fef0c44fb2bb590cb1c2a15520"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46544
I20260929 14:52:10.175503 22680 raft_consensus.cc:482] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:52:10.175630 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:10.178056 22680 raft_consensus.cc:504] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.180589 22857 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.180951 22857 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5837ee2faef4cd49b63a11715bebdd7, State: Initialized, Role: FOLLOWER
I20260929 14:52:10.181383 22857 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.183173 22857 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:10.183281 22754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b25915fef0c44fb2bb590cb1c2a15520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4874aa4e2913481eb32959a53bf7f332"
I20260929 14:52:10.183566 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:10.185348 22754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25915fef0c44fb2bb590cb1c2a15520 in term 1.
I20260929 14:52:10.185860 22656 leader_election.cc:302] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4874aa4e2913481eb32959a53bf7f332, b25915fef0c44fb2bb590cb1c2a15520; no voters:
I20260929 14:52:10.186211 22866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:10.186301 22680 leader_election.cc:288] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [CANDIDATE]: Term 1 election: Requested vote from peers 4874aa4e2913481eb32959a53bf7f332 (127.0.98.61:37273), b5837ee2faef4cd49b63a11715bebdd7 (127.0.98.60:45381)
I20260929 14:52:10.186489 22866 raft_consensus.cc:686] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [term 1 LEADER]: Becoming Leader. State: Replica: b25915fef0c44fb2bb590cb1c2a15520, State: Running, Role: LEADER
I20260929 14:52:10.186890 22866 consensus_queue.cc:237] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } }
I20260929 14:52:10.187054 22831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b25915fef0c44fb2bb590cb1c2a15520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b5837ee2faef4cd49b63a11715bebdd7"
I20260929 14:52:10.187247 22831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:10.188726 22831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25915fef0c44fb2bb590cb1c2a15520 in term 1.
I20260929 14:52:10.188908 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b25915fef0c44fb2bb590cb1c2a15520. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.189014 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: This master's current role is: LEADER
W20260929 14:52:10.189993 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:10.190083 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:10.202016 22908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:10.204759 22908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:10.207782 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.208505 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.214805 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.202775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.98.61:37273)
0929 14:52:10.202921 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.202925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.203480 (+ 555us) server_negotiation.cc:407] Connection header received
0929 14:52:10.203731 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.203735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.203806 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.203877 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.205082 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.205918 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.207165 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.207527 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.207603 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.210089 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.210094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.210097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.210129 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.211943 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.212617 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.212623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.212628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.212842 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.213205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.213208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.213210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.213457 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.213699 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.214336 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.214493 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:52:10.229081 22831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [term 1 FOLLOWER]: Refusing update from remote peer b25915fef0c44fb2bb590cb1c2a15520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:52:10.229777 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.202587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.98.62:33811)
0929 14:52:10.202777 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.202782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.207557 (+ 4775us) server_negotiation.cc:407] Connection header received
0929 14:52:10.208272 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.208275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.208326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.208378 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.213526 (+ 5148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.214300 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.217515 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.217900 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.217990 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.224734 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.224740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.224743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.224784 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.226970 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.227907 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.227911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.227913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.228002 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.228384 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.228388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.228390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.228865 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.229013 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.229416 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.229587 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 14:52:10.230738 22905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:52:10.234731 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b25915fef0c44fb2bb590cb1c2a15520. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.234862 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.236951 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:10.203423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.0.98.60:45381)
0929 14:52:10.203559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.203564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.209252 (+ 5688us) server_negotiation.cc:407] Connection header received
0929 14:52:10.209431 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.209435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.209486 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.209542 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.210423 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.211250 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.213886 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:10.214535 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:10.214688 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:10.225691 (+ 11003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:10.225695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:10.225698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:10.225730 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:10.230574 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.235508 (+ 4934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.235513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.235516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.235612 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:10.235913 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:10.235916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:10.235918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:10.236156 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:10.236312 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:10.236674 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 14:52:10.236801 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:52:10.238672 22754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [term 1 FOLLOWER]: Refusing update from remote peer b25915fef0c44fb2bb590cb1c2a15520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:52:10.239243 22905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:52:10.242883 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b25915fef0c44fb2bb590cb1c2a15520. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.243031 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.249145 22905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.249296 22905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:10.249617 22916 mvcc.cc:205] Tried to move back new op lower bound from 7334680699800457216 to 7334680699649597440. Current Snapshot: MvccSnapshot[applied={T|T < 7334680699800457216}]
I20260929 14:52:10.250695 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.250803 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.251081 22908 catalog_manager.cc:1347] Generated new cluster ID: 6fba715d79f3462c97c0401baeeb3026
I20260929 14:52:10.251137 22908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:10.265089 22917 mvcc.cc:205] Tried to move back new op lower bound from 7334680699800457216 to 7334680699649597440. Current Snapshot: MvccSnapshot[applied={T|T < 7334680699800457216}]
I20260929 14:52:10.273007 22866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.273181 22866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:10.276809 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b25915fef0c44fb2bb590cb1c2a15520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b25915fef0c44fb2bb590cb1c2a15520" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33811 } } peers { permanent_uuid: "4874aa4e2913481eb32959a53bf7f332" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37273 } } peers { permanent_uuid: "b5837ee2faef4cd49b63a11715bebdd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45381 } } }
I20260929 14:52:10.276921 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:10.303411 22908 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:10.304796 22908 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:10.328184 22908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b25915fef0c44fb2bb590cb1c2a15520: Generated new TSK 0
I20260929 14:52:10.329299 22908 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:52:10.370471 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.366663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.0.98.61:37273)
0929 14:52:10.366839 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.366844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.369173 (+ 2329us) server_negotiation.cc:407] Connection header received
0929 14:52:10.369412 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.369417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.369478 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.369540 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.370287 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50342: BlockingRecv error: recv got EOF from 127.0.0.1:50342 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 14:52:10.372212 22860 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.367602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.0.98.60:45381)
0929 14:52:10.367796 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.367802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.371124 (+ 3322us) server_negotiation.cc:407] Connection header received
0929 14:52:10.371315 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.371321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.371379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.371434 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.372028 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49270: BlockingRecv error: recv got EOF from 127.0.0.1:49270 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 14:52:10.374233 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.368392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.98.62:33811)
0929 14:52:10.368542 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.368547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.372935 (+ 4388us) server_negotiation.cc:407] Connection header received
0929 14:52:10.373396 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.373400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.373466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.373525 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.374089 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38436: BlockingRecv error: recv got EOF from 127.0.0.1:38436 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 14:52:10.379675 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.377951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.98.61:37273)
0929 14:52:10.378071 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.378076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.378595 (+ 519us) server_negotiation.cc:407] Connection header received
0929 14:52:10.378860 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.378865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.378964 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.379022 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.379555 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50358: BlockingRecv error: recv got EOF from 127.0.0.1:50358 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 14:52:10.385702 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.383661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.61:37273)
0929 14:52:10.383867 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.383874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.384527 (+ 653us) server_negotiation.cc:407] Connection header received
0929 14:52:10.384853 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.384857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.384918 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.384996 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.385572 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50370: BlockingRecv error: recv got EOF from 127.0.0.1:50370 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260929 14:52:10.392234 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.390316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.98.61:37273)
0929 14:52:10.390441 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.390445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.391067 (+ 622us) server_negotiation.cc:407] Connection header received
0929 14:52:10.391289 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.391293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.391357 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.391419 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.392047 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50384: BlockingRecv error: recv got EOF from 127.0.0.1:50384 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:52:10.400887 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.398843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.0.98.61:37273)
0929 14:52:10.399026 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.399030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.399690 (+ 660us) server_negotiation.cc:407] Connection header received
0929 14:52:10.399936 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.399940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.400002 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.400065 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.400729 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50396: BlockingRecv error: recv got EOF from 127.0.0.1:50396 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 14:52:10.413955 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.412001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.0.98.61:37273)
0929 14:52:10.412140 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.412145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.412805 (+ 660us) server_negotiation.cc:407] Connection header received
0929 14:52:10.413036 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.413041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.413105 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.413167 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.413817 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50398: BlockingRecv error: recv got EOF from 127.0.0.1:50398 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 14:52:10.434731 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.433087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.98.61:37273)
0929 14:52:10.433250 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.433255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.433624 (+ 369us) server_negotiation.cc:407] Connection header received
0929 14:52:10.433793 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.433797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.433878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.433992 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.434579 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50412: BlockingRecv error: recv got EOF from 127.0.0.1:50412 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 14:52:10.471406 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.469873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.61:37273)
0929 14:52:10.470100 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.470106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.470255 (+ 149us) server_negotiation.cc:407] Connection header received
0929 14:52:10.470511 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.470514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.470570 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.470672 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.471265 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50414: BlockingRecv error: recv got EOF from 127.0.0.1:50414 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 14:52:10.539255 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.537938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.0.98.61:37273)
0929 14:52:10.538075 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.538078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.538360 (+ 282us) server_negotiation.cc:407] Connection header received
0929 14:52:10.538476 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.538478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.538530 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.538635 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.539137 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50422: BlockingRecv error: recv got EOF from 127.0.0.1:50422 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 14:52:10.671314 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.670079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.0.98.61:37273)
0929 14:52:10.670289 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.670296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.670421 (+ 125us) server_negotiation.cc:407] Connection header received
0929 14:52:10.670548 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.670551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.670609 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.670718 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.671190 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50438: BlockingRecv error: recv got EOF from 127.0.0.1:50438 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 14:52:10.931408 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:10.930085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.0.98.61:37273)
0929 14:52:10.930296 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:10.930301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:10.930456 (+ 155us) server_negotiation.cc:407] Connection header received
0929 14:52:10.930603 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:10.930606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:10.930667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:10.930754 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:10.931246 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50446: BlockingRecv error: recv got EOF from 127.0.0.1:50446 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 14:52:11.173828 22890 catalog_manager.cc:1259] Loaded cluster ID: 6fba715d79f3462c97c0401baeeb3026
I20260929 14:52:11.173985 22890 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: loading cluster ID for follower catalog manager: success
I20260929 14:52:11.175951 22890 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: acquiring CA information for follower catalog manager: success
I20260929 14:52:11.177340 22890 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4874aa4e2913481eb32959a53bf7f332: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:52:11.191421 22904 catalog_manager.cc:1259] Loaded cluster ID: 6fba715d79f3462c97c0401baeeb3026
I20260929 14:52:11.191536 22904 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: loading cluster ID for follower catalog manager: success
I20260929 14:52:11.193213 22904 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: acquiring CA information for follower catalog manager: success
I20260929 14:52:11.194346 22904 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5837ee2faef4cd49b63a11715bebdd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:52:11.450457 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:11.446036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.0.98.61:37273)
0929 14:52:11.446341 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:11.446346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:11.446371 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:52:11.446519 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:11.446522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:11.446690 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:11.446842 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:11.447131 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.447743 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.448725 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.448983 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.449074 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:11.449409 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:11.450014 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:11.450124 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:11.450224 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:52:11.450287 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":113,"threads_started":1}
I20260929 14:52:11.459043 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:11.454463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.98.60:45381)
0929 14:52:11.454827 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:11.454833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:11.454849 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:52:11.454956 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:11.454959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:11.455105 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:11.455229 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:11.455479 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.456079 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.457152 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.457404 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.457515 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:11.457851 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:11.458466 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:11.458582 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:11.458705 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 14:52:11.458766 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260929 14:52:11.467612 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:11.462632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.98.61:37273)
0929 14:52:11.462780 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:11.462785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:11.463133 (+ 348us) server_negotiation.cc:407] Connection header received
0929 14:52:11.463398 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:11.463402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:11.463465 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:11.463544 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:11.463894 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.464789 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.466098 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.466442 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.466567 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:11.466984 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:11.467082 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:11.467169 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:11.467380 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 14:52:11.467456 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:52:11.468338 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:11.463140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.0.98.60:45381)
0929 14:52:11.463295 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:11.463300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:11.463486 (+ 186us) server_negotiation.cc:407] Connection header received
0929 14:52:11.463603 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:11.463606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:11.463654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:11.463737 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:11.464084 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.464705 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.466982 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.467345 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.467420 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:11.467821 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:11.467913 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:11.467965 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:11.468136 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:52:11.468219 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:52:11.468550 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:11.462629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.0.98.62:33811)
0929 14:52:11.463344 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:11.463352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:11.463368 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:52:11.463470 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:11.463473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:11.463665 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:11.463784 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:11.464144 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.464979 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.466225 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:11.466594 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:11.466712 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:11.467162 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:11.467699 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:11.467831 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:11.468165 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 14:52:11.468291 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":94,"threads_started":1}
I20260929 14:52:11.472244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22638
I20260929 14:52:11.481142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22713
I20260929 14:52:11.489087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 22789
2026-09-29T14:52:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3227 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:52:11 dist-test-slave-5hd1 krb5kdc[23005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:52:11 dist-test-slave-5hd1 krb5kdc[23005](info): set up 2 sockets
Sep 29 14:52:11 dist-test-slave-5hd1 krb5kdc[23005](info): commencing operation
krb5kdc: starting...
W20260929 14:52:12.558251 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/test-user.keytab.
Sep 29 14:52:12 dist-test-slave-5hd1 krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:52:12.699868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:52:12.826356 23018 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:12.826728 23018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:12.826756 23018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:12.826799 23018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:12.831367 23018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:12.831452 23018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:12.831472 23018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:12.831490 23018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:12.831507 23018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:12.837421 23018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:12.838773 23018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:12.839967 23018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:12.850139 23026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:12.850142 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:12.850142 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:12.850293 23018 server_base.cc:1034] running on GCE node
I20260929 14:52:12.851280 23018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:12.852687 23018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:12.853871 23018 hybrid_clock.cc:648] HybridClock initialized: now 1790693532853839 us; error 44 us; skew 500 ppm
Sep 29 14:52:12 dist-test-slave-5hd1 krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:12.857309 23018 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:52:12.858385 23018 webserver.cc:460] Webserver started at http://127.0.98.62:38233/ using document root <none> and password file <none>
I20260929 14:52:12.859091 23018 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:12.859163 23018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:12.859422 23018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:12.861907 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data/instance:
uuid: "daad5813d4cd47899eab918c17186796"
format_stamp: "Formatted at 2026-09-29 14:52:12 on dist-test-slave-5hd1"
server_key: "c1fe328a95dfbe81bc59123ee8329cf8"
server_key_iv: "7dfd08f755785946d609bab6e756f1e2"
server_key_version: "encryptionkey@0"
I20260929 14:52:12.862564 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance:
uuid: "daad5813d4cd47899eab918c17186796"
format_stamp: "Formatted at 2026-09-29 14:52:12 on dist-test-slave-5hd1"
server_key: "c1fe328a95dfbe81bc59123ee8329cf8"
server_key_iv: "7dfd08f755785946d609bab6e756f1e2"
server_key_version: "encryptionkey@0"
I20260929 14:52:12.867007 23018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:52:12.869979 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:12.871529 23018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:52:12.871690 23018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal
uuid: "daad5813d4cd47899eab918c17186796"
format_stamp: "Formatted at 2026-09-29 14:52:12 on dist-test-slave-5hd1"
server_key: "c1fe328a95dfbe81bc59123ee8329cf8"
server_key_iv: "7dfd08f755785946d609bab6e756f1e2"
server_key_version: "encryptionkey@0"
I20260929 14:52:12.871822 23018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:12.906395 23018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:12.910449 23018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:12.910701 23018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:12.919667 23018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37055
I20260929 14:52:12.920063 23085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37055 every 8 connection(s)
I20260929 14:52:12.920941 23018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:52:12.924196 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:12.927665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 23018
I20260929 14:52:12.927852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-0/wal/instance
I20260929 14:52:12.928264 392 external_mini_cluster.cc:1428] Setting key ebd418a0bff594ab96733814c218b6d2
I20260929 14:52:12.928335 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:52:12 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:12.949180 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:12.931482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.0.1:46450)
0929 14:52:12.932240 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:12.932312 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:12.933758 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:12.934682 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:12.934691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:12.935313 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:12.935823 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:12.935847 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:12.937237 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:12.937242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:12.938714 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:12.938731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:12.939613 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:12.939616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:12.939742 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:12.940483 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:12.940501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:12.942730 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:12.945472 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:12.945482 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:12.945493 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:12.945803 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:12.946234 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:12.946236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:12.946237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:12.946355 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:12.947244 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:12.947251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:12.947416 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 14:52:12.947589 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 14:52:12.947945 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":180,"threads_started":1}
I20260929 14:52:12.949187 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:12.931482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.98.62:37055)
0929 14:52:12.932474 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:12.932488 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:12.932549 (+ 61us) server_negotiation.cc:407] Connection header received
0929 14:52:12.934148 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:12.934154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:12.934394 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:12.934600 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:12.936021 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:12.937062 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:12.938902 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:12.939404 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:12.939606 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:12.942946 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:12.942953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:12.942965 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:12.942991 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:12.945204 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:12.945946 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:12.945956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:12.945962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:12.946121 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:12.946465 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:12.946472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:12.946477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:12.947111 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:12.947284 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:12.947744 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 14:52:12.948089 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":560,"spinlock_wait_cycles":65792,"thread_start_us":189,"threads_started":1}
I20260929 14:52:12.952014 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:12.952965 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:12.952681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50816)
0929 14:52:12.952819 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:12.952901 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 14:52:12.953270 23035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
I20260929 14:52:12.954384 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:12.954162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50822)
0929 14:52:12.954279 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:12.954327 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 14:52:12.954717 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:52:12.959550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39133
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:52:13.004024 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } attempt: 1
I20260929 14:52:13.005352 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.004942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50828)
0929 14:52:13.005162 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.005250 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 14:52:13.006188 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.005962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50840)
0929 14:52:13.006078 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.006134 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:52:13.006321 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
W20260929 14:52:13.074056 23093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:13.074419 23093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:13.074492 23093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:13.074591 23093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 14:52:13.074554 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } attempt: 2
I20260929 14:52:13.075606 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.075212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50848)
0929 14:52:13.075390 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.075491 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:52:13.076619 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.076275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50856)
0929 14:52:13.076421 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.076520 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 14:52:13.076845 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39133: connect: Connection refused (error 111)
W20260929 14:52:13.079252 23093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:13.079305 23093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:13.079324 23093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:13.079341 23093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:13.079361 23093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:13.084683 23093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:13.085757 23093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:13.086838 23093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:13.096016 23101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:13.096051 23098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:13.096006 23099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:13.096587 23093 server_base.cc:1034] running on GCE node
I20260929 14:52:13.097275 23093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:13.098677 23093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:13.099881 23093 hybrid_clock.cc:648] HybridClock initialized: now 1790693533099856 us; error 42 us; skew 500 ppm
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:13.103243 23093 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:52:13.104420 23093 webserver.cc:460] Webserver started at http://127.0.98.61:39199/ using document root <none> and password file <none>
I20260929 14:52:13.105113 23093 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:13.105212 23093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:13.105481 23093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:13.107933 23093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data/instance:
uuid: "5d959b8e33934b258d7f1e520b097af0"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "262633f91c8658780465f42f95fc535c"
server_key_iv: "468a11fd28cc6626d1aab99923a55ef5"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.108629 23093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d959b8e33934b258d7f1e520b097af0"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "262633f91c8658780465f42f95fc535c"
server_key_iv: "468a11fd28cc6626d1aab99923a55ef5"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.113212 23093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:52:13.116499 23108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:13.118191 23093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:52:13.118379 23093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal
uuid: "5d959b8e33934b258d7f1e520b097af0"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "262633f91c8658780465f42f95fc535c"
server_key_iv: "468a11fd28cc6626d1aab99923a55ef5"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.118527 23093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:13.149920 23093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:13.154109 23093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:13.154346 23093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:13.163602 23093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39133
I20260929 14:52:13.163975 23160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39133 every 8 connection(s)
I20260929 14:52:13.164815 23093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:52:13.167392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 23093
I20260929 14:52:13.167578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-1/wal/instance
I20260929 14:52:13.167918 392 external_mini_cluster.cc:1428] Setting key 0c0c19d336ac72522e4fde05bfd67976
I20260929 14:52:13.167991 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:13.168038 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } attempt: 3
I20260929 14:52:13.174257 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 14:52:13.183897 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.168754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50860)
0929 14:52:13.168979 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.168996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.169095 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.172728 (+ 3633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.172732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.172982 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.173168 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.173176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.174633 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.174639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.175286 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.175296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.176146 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.176150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.176296 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.177092 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.177110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.178752 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.181474 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.181479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.181481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.181874 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.182342 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.182345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.182348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.182449 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.183277 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.183281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.183394 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.183519 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.183700 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:13.186311 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.169842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.0.98.61:39133)
0929 14:52:13.170411 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.170438 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.170498 (+ 60us) server_negotiation.cc:407] Connection header received
0929 14:52:13.171822 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.171845 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.172354 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.172781 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.173302 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.174477 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.175414 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.176009 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.176215 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.178947 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.178974 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.178991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.179028 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.181315 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.182039 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.182058 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.182071 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.182191 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.182546 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.182553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.182557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.183160 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.183320 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.183684 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.184772 (+ 1088us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":177,"threads_started":1}
I20260929 14:52:13.189164 23086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:52:13.190039 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.189697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60238)
0929 14:52:13.189908 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.189963 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 14:52:13.190527 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.176888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.0.1:46462)
0929 14:52:13.177523 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.177600 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.177826 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.178230 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.178239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.178788 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.179052 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.179065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.180251 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.180255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.181295 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.181308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.182259 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.182267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.182459 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.183204 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.183228 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.185250 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.188348 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.188355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.188360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.188683 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.189095 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.189100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.189102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.189202 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.189779 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.189786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.189924 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.190102 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.190319 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":176,"threads_started":1}
I20260929 14:52:13.190652 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.176939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.98.62:37055)
0929 14:52:13.177158 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.177167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.177702 (+ 535us) server_negotiation.cc:407] Connection header received
0929 14:52:13.177974 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.177977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.178032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.178127 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.179206 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.179974 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.181474 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.181970 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.182089 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.185425 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.185429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.185432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.185457 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.188207 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.188823 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.188830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.188834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.188958 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.189327 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.189330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.189333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.189638 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.189764 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.190219 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.190422 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 14:52:13.191335 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.190601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60254)
0929 14:52:13.191184 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.191235 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":134,"threads_started":1}
W20260929 14:52:13.191936 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.192142 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:52:13.201994 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.192718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50866)
0929 14:52:13.192807 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.192821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.192897 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.193248 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.193251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.193406 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.193524 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.193530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.194489 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.194492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.195128 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.195137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.195679 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.195683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.195806 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.196271 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.196282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.197839 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.200144 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.200147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.200149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.200510 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.200817 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.200820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.200821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.200899 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.201435 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.201437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.201543 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.201645 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.201803 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260929 14:52:13.202109 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.192856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.98.61:39133)
0929 14:52:13.192984 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.192989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.193013 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:52:13.193048 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.193050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.193089 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.193171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.193741 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.194357 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.195263 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.195510 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.195662 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.198003 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.198008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.198018 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.198045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.199926 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.200636 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.200638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.200640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.200721 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.201059 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.201063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.201065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.201333 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.201427 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.201751 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.201909 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":46}
I20260929 14:52:13.203142 23161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:52:13.203970 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.203688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60266)
0929 14:52:13.203826 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.203916 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 14:52:13.204404 23111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.204867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
/tmp/dist-test-taskHiGqqz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34943
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:52:13.205500 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.205293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60270)
0929 14:52:13.205403 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.205452 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 14:52:13.205756 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.223141 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 1
I20260929 14:52:13.224447 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.223955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60272)
0929 14:52:13.224229 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.224333 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 14:52:13.225308 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.225033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60274)
0929 14:52:13.225204 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.225259 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 14:52:13.225452 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.255070 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 1
I20260929 14:52:13.256822 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.256450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60278)
0929 14:52:13.256649 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.256743 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 14:52:13.257689 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.257484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60290)
0929 14:52:13.257586 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.257640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 14:52:13.257812 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.300652 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 2
I20260929 14:52:13.301600 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.301281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60292)
0929 14:52:13.301452 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.301521 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 14:52:13.302379 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.302164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60302)
0929 14:52:13.302275 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.302329 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 14:52:13.302553 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.325999 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 2
I20260929 14:52:13.327072 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.326692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60312)
0929 14:52:13.326903 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.326984 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 14:52:13.327925 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.327722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60328)
0929 14:52:13.327824 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.327877 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 14:52:13.328042 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
W20260929 14:52:13.336630 23169 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:52:13.336985 23169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:52:13.337045 23169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:52:13.337144 23169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:52:13.341581 23169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:52:13.341683 23169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:52:13.341718 23169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:52:13.341743 23169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:52:13.341789 23169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:52:13.347805 23169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37055,127.0.98.61:39133,127.0.98.60:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:52:13.349272 23169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:52:13.350497 23169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:52:13.360633 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:13.360636 23175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:52:13.360636 23174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:52:13.361243 23169 server_base.cc:1034] running on GCE node
I20260929 14:52:13.362012 23169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:52:13.363538 23169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:52:13.364779 23169 hybrid_clock.cc:648] HybridClock initialized: now 1790693533364740 us; error 45 us; skew 500 ppm
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:52:13.368455 23169 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:52:13.369762 23169 webserver.cc:460] Webserver started at http://127.0.98.60:42537/ using document root <none> and password file <none>
I20260929 14:52:13.370498 23169 fs_manager.cc:359] Metadata directory not provided
I20260929 14:52:13.370615 23169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:52:13.370894 23169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:52:13.373621 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data/instance:
uuid: "286b7eed13da471283ec81ef84ddd7b6"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "4d41675d7f696d2e6ed80f4eaac23f40"
server_key_iv: "0a4b406740ea68f22f25c7131a43aabb"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.374333 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal/instance:
uuid: "286b7eed13da471283ec81ef84ddd7b6"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "4d41675d7f696d2e6ed80f4eaac23f40"
server_key_iv: "0a4b406740ea68f22f25c7131a43aabb"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.379346 23169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 14:52:13.382700 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:52:13.384433 23169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 14:52:13.384619 23169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal
uuid: "286b7eed13da471283ec81ef84ddd7b6"
format_stamp: "Formatted at 2026-09-29 14:52:13 on dist-test-slave-5hd1"
server_key: "4d41675d7f696d2e6ed80f4eaac23f40"
server_key_iv: "0a4b406740ea68f22f25c7131a43aabb"
server_key_version: "encryptionkey@0"
I20260929 14:52:13.384774 23169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:52:13.401742 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 3
I20260929 14:52:13.402823 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.402459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60334)
0929 14:52:13.402659 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.402741 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 14:52:13.403622 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.403425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60348)
0929 14:52:13.403522 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.403576 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:52:13.403755 23086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.412775 23169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:52:13.416678 23169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:52:13.416915 23169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:52:13.419250 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 3
I20260929 14:52:13.420282 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.419894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60360)
0929 14:52:13.420073 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.420143 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 14:52:13.421092 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.420841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60374)
0929 14:52:13.420973 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.421024 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 14:52:13.421304 23161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34943: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34943: connect: Connection refused (error 111)
I20260929 14:52:13.426482 23169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34943
I20260929 14:52:13.426824 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34943 every 8 connection(s)
I20260929 14:52:13.427819 23169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:52:13.431216 23237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:52:13.433511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu as pid 23169
I20260929 14:52:13.433796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiGqqz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790693238534145-392-0/minicluster-data/master-2/wal/instance
I20260929 14:52:13.434213 392 external_mini_cluster.cc:1428] Setting key 676b4d775543470444f2256480e8156a
I20260929 14:52:13.435664 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:52:13.449716 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.437898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.62:37055)
0929 14:52:13.438085 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.438090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.438396 (+ 306us) server_negotiation.cc:407] Connection header received
0929 14:52:13.438597 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.438600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.438651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.438732 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.439938 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.440570 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.441711 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.441970 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.442119 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.444846 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.444850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.444852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.444884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.446850 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.447495 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.447501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.447503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.447619 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.448097 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.448100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.448101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.448429 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.448557 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.449178 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.449332 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:52:13.450532 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.436123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.0.98.60:34943)
0929 14:52:13.436744 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.436765 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.436815 (+ 50us) server_negotiation.cc:407] Connection header received
0929 14:52:13.437618 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.437635 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.438074 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.438471 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.439369 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.440669 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.441609 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.441983 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.442169 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.445476 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.445487 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.445510 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.445546 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.447789 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.448393 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.448400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.448402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.448500 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.448805 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.448807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.448808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.449128 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.449354 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.449741 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.449934 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":439,"thread_start_us":173,"threads_started":1}
I20260929 14:52:13.450531 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.437816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37055 (local address 127.0.0.1:46470)
0929 14:52:13.438264 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.438319 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.438481 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.438921 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.438925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.439274 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.439743 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.439757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.440709 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.440715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.441485 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.441494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.442161 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.442165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.442274 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.443016 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.443032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.444633 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.446999 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.447008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.447019 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.447354 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.447749 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.447752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.447753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.447972 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.448569 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.448578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.448927 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.449093 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.449464 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":42,"thread_start_us":165,"threads_started":1}
I20260929 14:52:13.452060 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:52:13.462337 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.452877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.98.61:39133)
0929 14:52:13.453061 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.453067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.453088 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:52:13.453142 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.453145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.453202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.453305 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.454334 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.454991 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.455832 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.456071 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.456185 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.458731 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.458735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.458737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.458793 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.460389 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.460855 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.460859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.460861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.460946 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.461240 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.461245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.461247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.461448 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.461727 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.462027 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.462161 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 14:52:13.462337 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.452736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39133 (local address 127.0.0.1:50872)
0929 14:52:13.452939 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.452958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.453049 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.453885 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.453887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.454083 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.454195 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.454200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.455133 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.455139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.455706 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.455714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.456211 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.456215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.456337 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.456898 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.456919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.458530 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.460514 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.460517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.460519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.460745 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.461053 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.461057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.461059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.461145 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.461709 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.461713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.461843 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.461968 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.462142 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 14:52:13.462846 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.453306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.98.62:37055)
0929 14:52:13.453476 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.453479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.453489 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:52:13.453518 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.453520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.453675 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.453761 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.454401 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.454939 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.455700 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.455921 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.456032 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.459376 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.459379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.459381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.459407 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.461034 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.461471 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.461477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.461479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.461556 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.461843 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.461848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.461850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.462027 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.462235 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.462372 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.462725 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 14:52:13.463904 23237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:52:13.474400 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.464550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60406)
0929 14:52:13.464863 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.464876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.464969 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.465233 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.465239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.465432 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.465573 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.465580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.466650 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.466656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.467540 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.467560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.468272 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.468276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.468365 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.468966 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.468985 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.470506 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.472611 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.472618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.472621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.472855 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.473158 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.473161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.473164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.473267 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.473785 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.473790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.473899 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.474017 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.474155 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260929 14:52:13.474404 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.464781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.98.60:34943)
0929 14:52:13.464947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.464954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.464970 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:52:13.465066 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.465069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.465111 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.465181 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.465689 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.466489 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.467719 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.468106 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.468238 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.470661 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.470665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.470668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.470695 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.472434 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.472972 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.472974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.472976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.473057 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.473390 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.473393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.473395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.473643 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.473780 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.474093 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.474240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:52:13.475756 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.464459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.0.98.61:39133)
0929 14:52:13.464567 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.464571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.464637 (+ 66us) server_negotiation.cc:407] Connection header received
0929 14:52:13.464794 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.464797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.464878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.464970 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.465806 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.466553 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.467720 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.468079 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.468238 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.471279 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.471284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.471287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.471323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.473202 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.473816 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.473824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.473825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.473914 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.474264 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.474268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.474271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.474517 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.475415 (+ 898us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.475503 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.475638 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:52:13.478560 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: Bootstrap starting.
I20260929 14:52:13.479851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:52:13.481083 23237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:13.482023 23237 log.cc:824] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:13.481348 23060 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "daad5813d4cd47899eab918c17186796"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46478
I20260929 14:52:13.484690 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: No bootstrap required, opened a new log
I20260929 14:52:13.487564 23237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.487763 23237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286b7eed13da471283ec81ef84ddd7b6, State: Initialized, Role: FOLLOWER
I20260929 14:52:13.488068 23237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.488756 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.488970 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.489382 23237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:52:13.494622 23254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:13.494704 23254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:13.493846 23169 master_runner.cc:186] Error getting master registration for 127.0.98.62:37055: OK, instance_id { permanent_uuid: "daad5813d4cd47899eab918c17186796" instance_seqno: 1790693532917169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:52:13.498631 23169 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:52:13.564531 23161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 4
I20260929 14:52:13.567926 23086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } attempt: 4
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:52:13 dist-test-slave-5hd1 krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:52:13.577072 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.565345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.98.60:34943)
0929 14:52:13.565496 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.565509 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.565676 (+ 167us) server_negotiation.cc:407] Connection header received
0929 14:52:13.565856 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.565859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.565911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.566032 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.566665 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.567468 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.568834 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.569199 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.569306 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.572138 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.572144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.572149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.572248 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.574805 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.575354 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.575360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.575363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.575468 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.575829 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.575833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.575835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.576232 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.576357 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.576776 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.576901 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:52:13.577463 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.565283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60410)
0929 14:52:13.565535 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.565562 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.565748 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.566061 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.566064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.566338 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.566516 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.566523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.567611 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.567615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.568559 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.568570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.569385 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.569390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.569532 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.570240 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.570258 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.571890 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.574931 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.574936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.574940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.575225 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.575567 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.575597 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.575601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.575702 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.576378 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.576404 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.576547 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.576708 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.576877 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 14:52:13.580871 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.568454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34943 (local address 127.0.0.1:60424)
0929 14:52:13.568679 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 14:52:13.568698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:52:13.568799 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:52:13.569406 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:52:13.569409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:52:13.569682 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:52:13.569862 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.569872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.571136 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.571140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.572051 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:52:13.572061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.572633 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.572637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:52:13.572738 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.573165 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:52:13.573190 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:52:13.574970 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:52:13.578917 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.578921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.578924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.579242 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.579582 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:52:13.579585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:52:13.579586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:52:13.579673 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:52:13.580228 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:52:13.580231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:52:13.580386 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 14:52:13.580500 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:52:13.580668 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260929 14:52:13.580926 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.568507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.98.60:34943)
0929 14:52:13.569134 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.569144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.569158 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:52:13.569198 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.569201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.569247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.570016 (+ 769us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.570145 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.570967 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.572271 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.572508 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.572627 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.575168 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.575172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.575175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.575202 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.578762 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.579374 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.579377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.579379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.579476 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.579782 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.579786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.579788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.580039 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.580246 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.580601 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.580732 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":160,"threads_started":1}
I20260929 14:52:13.581453 23161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: Bootstrap starting.
I20260929 14:52:13.583948 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796: Bootstrap starting.
I20260929 14:52:13.584342 23161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:13.585256 23161 log.cc:824] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:13.586452 23086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796: Neither blocks nor log segments found. Creating new log.
I20260929 14:52:13.587316 23086 log.cc:824] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796: Log is configured to *not* fsync() on all Append() calls
I20260929 14:52:13.588450 23161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: No bootstrap required, opened a new log
I20260929 14:52:13.589942 23086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796: No bootstrap required, opened a new log
I20260929 14:52:13.592460 23161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.592690 23161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d959b8e33934b258d7f1e520b097af0, State: Initialized, Role: FOLLOWER
I20260929 14:52:13.593017 23161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.593501 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.593695 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.594301 23161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:13.594360 23086 raft_consensus.cc:348] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.594655 23086 raft_consensus.cc:729] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daad5813d4cd47899eab918c17186796, State: Initialized, Role: FOLLOWER
I20260929 14:52:13.595008 23086 consensus_queue.cc:260] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.595628 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.595867 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.596777 23086 sys_catalog.cc:564] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:52:13.596835 23060 raft_consensus.cc:482] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:52:13.596977 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:13.598333 23060 raft_consensus.cc:504] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
W20260929 14:52:13.599797 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:52:13.599931 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:52:13.600277 23135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daad5813d4cd47899eab918c17186796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d959b8e33934b258d7f1e520b097af0"
I20260929 14:52:13.600508 23135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:13.601641 23135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daad5813d4cd47899eab918c17186796 in term 1.
I20260929 14:52:13.598327 23093 master_runner.cc:186] Error getting master registration for 127.0.98.62:37055: OK, instance_id { permanent_uuid: "daad5813d4cd47899eab918c17186796" instance_seqno: 1790693532917169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:52:13.602046 23060 leader_election.cc:288] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [CANDIDATE]: Term 1 election: Requested vote from peers 5d959b8e33934b258d7f1e520b097af0 (127.0.98.61:39133), 286b7eed13da471283ec81ef84ddd7b6 (127.0.98.60:34943)
I20260929 14:52:13.602119 23035 leader_election.cc:302] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d959b8e33934b258d7f1e520b097af0, daad5813d4cd47899eab918c17186796; no voters:
I20260929 14:52:13.602394 23263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:52:13.602850 23263 raft_consensus.cc:686] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [term 1 LEADER]: Becoming Leader. State: Replica: daad5813d4cd47899eab918c17186796, State: Running, Role: LEADER
I20260929 14:52:13.602977 23211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daad5813d4cd47899eab918c17186796" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "286b7eed13da471283ec81ef84ddd7b6"
I20260929 14:52:13.603199 23263 consensus_queue.cc:237] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } }
I20260929 14:52:13.603320 23211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:52:13.605093 23211 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daad5813d4cd47899eab918c17186796 in term 1.
I20260929 14:52:13.605571 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daad5813d4cd47899eab918c17186796. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.605769 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:13.605938 23093 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:52:13.606204 23281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:52:13.609280 23281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:52:13.625793 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.605463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.0.98.60:34943)
0929 14:52:13.605601 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.605605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.615790 (+ 10185us) server_negotiation.cc:407] Connection header received
0929 14:52:13.615871 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.615874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.615926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.616003 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.617285 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.618121 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.619372 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.619749 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.619832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.621977 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.621981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.621984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.622011 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.623889 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.624432 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.624436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.624438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.624524 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.624814 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.624818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.624820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.625022 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.625153 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.625485 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.625622 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:52:13.632746 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.609068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.0.98.61:39133)
0929 14:52:13.609200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.609206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.616896 (+ 7690us) server_negotiation.cc:407] Connection header received
0929 14:52:13.617076 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.617079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.617136 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.617187 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.618121 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.618893 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.619958 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.620294 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.620365 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.628365 (+ 8000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.628371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.628374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.628428 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.630594 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.631184 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.631187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.631189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.631281 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.631613 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.631616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.631618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.631870 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.631977 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.632383 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.632542 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:52:13.634389 23211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [term 1 FOLLOWER]: Refusing update from remote peer daad5813d4cd47899eab918c17186796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:52:13.634609 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:13.604771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.98.62:37055)
0929 14:52:13.604897 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.604904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.615376 (+ 10472us) server_negotiation.cc:407] Connection header received
0929 14:52:13.615672 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.615675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.615719 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.615823 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.616973 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.617826 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.619079 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:13.619416 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:13.619508 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:13.630140 (+ 10632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:52:13.630144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:52:13.630147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:52:13.630179 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:52:13.632034 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.632554 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.632557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.632559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.632642 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:52:13.633332 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:52:13.633338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:52:13.633343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:52:13.633583 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:52:13.633704 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:13.634346 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 14:52:13.634464 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:52:13.635200 23135 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [term 1 FOLLOWER]: Refusing update from remote peer daad5813d4cd47899eab918c17186796: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:52:13.636178 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:52:13.636857 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:52:13.640221 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daad5813d4cd47899eab918c17186796. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.640354 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.640642 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daad5813d4cd47899eab918c17186796. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.640763 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.648633 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.648761 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:13.649645 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.649758 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.650394 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.650503 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796 [sys.catalog]: This master's current role is: LEADER
I20260929 14:52:13.651412 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daad5813d4cd47899eab918c17186796" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daad5813d4cd47899eab918c17186796" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37055 } } peers { permanent_uuid: "5d959b8e33934b258d7f1e520b097af0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39133 } } peers { permanent_uuid: "286b7eed13da471283ec81ef84ddd7b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34943 } } }
I20260929 14:52:13.651528 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:52:13.651733 23296 mvcc.cc:205] Tried to move back new op lower bound from 7334680713753739264 to 7334680713642360832. Current Snapshot: MvccSnapshot[applied={T|T < 7334680713753739264}]
I20260929 14:52:13.653218 23297 mvcc.cc:205] Tried to move back new op lower bound from 7334680713753739264 to 7334680713642360832. Current Snapshot: MvccSnapshot[applied={T|T < 7334680713753739264}]
I20260929 14:52:13.658597 23281 catalog_manager.cc:1347] Generated new cluster ID: b998962a71bf474b823a53917a1ab33a
I20260929 14:52:13.658666 23281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:52:13.672801 23281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:52:13.674022 23281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:52:13.698356 23281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daad5813d4cd47899eab918c17186796: Generated new TSK 0
I20260929 14:52:13.699293 23281 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:52:13.737784 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.735410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.0.98.61:39133)
0929 14:52:13.735547 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.735552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.736218 (+ 666us) server_negotiation.cc:407] Connection header received
0929 14:52:13.736428 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.736432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.736482 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.736537 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.737637 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50902: BlockingRecv error: recv got EOF from 127.0.0.1:50902 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 14:52:13.739159 23090 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.735491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.98.62:37055)
0929 14:52:13.735658 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.735663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.737375 (+ 1712us) server_negotiation.cc:407] Connection header received
0929 14:52:13.737897 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.737901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.737965 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.738025 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.739030 (+ 1005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46496: BlockingRecv error: recv got EOF from 127.0.0.1:46496 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 14:52:13.739555 23256 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.736431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.98.60:34943)
0929 14:52:13.736562 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.736566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.737595 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 14:52:13.738181 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.738185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.738239 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.738291 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.739393 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60442: BlockingRecv error: recv got EOF from 127.0.0.1:60442 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:52:13.743305 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.98.61:39133)
0929 14:52:13.742080 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.742084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.742355 (+ 271us) server_negotiation.cc:407] Connection header received
0929 14:52:13.742516 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.742519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.742626 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.742717 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.743176 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50912: BlockingRecv error: recv got EOF from 127.0.0.1:50912 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 14:52:13.747430 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.746465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.0.98.61:39133)
0929 14:52:13.746615 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.746620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.746637 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:52:13.746694 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.746697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.746757 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.746859 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.747290 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50918: BlockingRecv error: recv got EOF from 127.0.0.1:50918 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 14:52:13.752126 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.751038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.0.98.61:39133)
0929 14:52:13.751155 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.751160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.751275 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:52:13.751392 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.751396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.751458 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.751567 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.751995 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50922: BlockingRecv error: recv got EOF from 127.0.0.1:50922 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:52:13.759529 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.758104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.0.98.61:39133)
0929 14:52:13.758240 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.758246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.758550 (+ 304us) server_negotiation.cc:407] Connection header received
0929 14:52:13.758701 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.758704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.758777 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.758884 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.759390 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50928: BlockingRecv error: recv got EOF from 127.0.0.1:50928 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:52:13.771450 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.769905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.0.98.61:39133)
0929 14:52:13.770039 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.770044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.770404 (+ 360us) server_negotiation.cc:407] Connection header received
0929 14:52:13.770541 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.770549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.770635 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.770748 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.771279 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50942: BlockingRecv error: recv got EOF from 127.0.0.1:50942 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 14:52:13.791230 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.789650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.0.98.61:39133)
0929 14:52:13.789795 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.789800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.790175 (+ 375us) server_negotiation.cc:407] Connection header received
0929 14:52:13.790333 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.790338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.790433 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.790548 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.791090 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50952: BlockingRecv error: recv got EOF from 127.0.0.1:50952 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 14:52:13.828442 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.826804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.0.98.61:39133)
0929 14:52:13.827027 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.827034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.827276 (+ 242us) server_negotiation.cc:407] Connection header received
0929 14:52:13.827433 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.827436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.827496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.827607 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.828310 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50960: BlockingRecv error: recv got EOF from 127.0.0.1:50960 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260929 14:52:13.896970 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:13.895437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.98.61:39133)
0929 14:52:13.895625 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:13.895631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:13.895844 (+ 213us) server_negotiation.cc:407] Connection header received
0929 14:52:13.896055 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:13.896061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:13.896118 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:13.896253 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:13.896846 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50970: BlockingRecv error: recv got EOF from 127.0.0.1:50970 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 14:52:14.029305 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:14.028036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.61:39133)
0929 14:52:14.028264 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.028269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.028342 (+ 73us) server_negotiation.cc:407] Connection header received
0929 14:52:14.028485 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.028487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.028536 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.028645 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:14.029144 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50982: BlockingRecv error: recv got EOF from 127.0.0.1:50982 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 14:52:14.289335 23164 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:52:14.287979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.98.61:39133)
0929 14:52:14.288149 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.288153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.288197 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:52:14.288272 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.288274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.288382 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.288555 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:52:14.289206 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50998: BlockingRecv error: recv got EOF from 127.0.0.1:50998 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:52:14.496282 23254 catalog_manager.cc:1259] Loaded cluster ID: b998962a71bf474b823a53917a1ab33a
I20260929 14:52:14.496379 23254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: loading cluster ID for follower catalog manager: success
I20260929 14:52:14.497994 23254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: acquiring CA information for follower catalog manager: success
I20260929 14:52:14.499112 23254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 286b7eed13da471283ec81ef84ddd7b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:52:14.601447 23271 catalog_manager.cc:1259] Loaded cluster ID: b998962a71bf474b823a53917a1ab33a
I20260929 14:52:14.601567 23271 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: loading cluster ID for follower catalog manager: success
I20260929 14:52:14.603278 23271 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: acquiring CA information for follower catalog manager: success
I20260929 14:52:14.604530 23271 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d959b8e33934b258d7f1e520b097af0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:52:14.808195 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:14.803812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.0.98.61:39133)
0929 14:52:14.804050 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.804054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.804229 (+ 175us) server_negotiation.cc:407] Connection header received
0929 14:52:14.804465 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.804469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.804602 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.804734 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:14.805045 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.805572 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.806639 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.806863 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.806947 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:14.807277 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:14.807731 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:14.807822 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:14.807909 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 14:52:14.807958 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260929 14:52:14.815776 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:14.811996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.98.60:34943)
0929 14:52:14.812307 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.812310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.812327 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:52:14.812480 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.812483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.812596 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.812713 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:14.812900 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.813452 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.814365 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.814593 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.814681 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:14.814962 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:14.815379 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:14.815474 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:14.815608 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 14:52:14.815655 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260929 14:52:14.823886 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:14.819431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.0.98.60:34943)
0929 14:52:14.819587 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.819592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.819876 (+ 284us) server_negotiation.cc:407] Connection header received
0929 14:52:14.819932 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.819935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.820008 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.820074 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:14.820522 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.821017 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822548 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.822869 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822957 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:14.823440 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:14.823514 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:14.823597 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:14.823693 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 14:52:14.823741 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:52:14.824069 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:14.819051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.0.98.61:39133)
0929 14:52:14.819184 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.819189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.819747 (+ 558us) server_negotiation.cc:407] Connection header received
0929 14:52:14.819861 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.819867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.819944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.820031 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:14.820343 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.821180 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822471 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.822827 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:14.823479 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:14.823574 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:14.823646 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:14.823796 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 14:52:14.823875 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:52:14.824496 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:52:14.819056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.98.62:37055)
0929 14:52:14.819734 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 14:52:14.819739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:52:14.819755 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:52:14.819850 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:52:14.819854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:52:14.819984 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:52:14.820122 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:52:14.820573 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.821371 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822594 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:52:14.822824 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:52:14.822947 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:52:14.823361 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:52:14.823792 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:52:14.823874 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:52:14.824230 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 14:52:14.824303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":79,"threads_started":1}
I20260929 14:52:14.834509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 23018
I20260929 14:52:14.842252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 23093
I20260929 14:52:14.849238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiGqqz/build/debug/bin/kudu with pid 23169
2026-09-29T14:52:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3360 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6587 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (296306 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST